{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "2 DL Multi-Layer NN for DigitRecognizer.ipynb",
      "version": "0.3.2",
      "provenance": [],
      "collapsed_sections": [
        "b6I1adl5gBQD",
        "xeMZR7ntgBQI",
        "LT78eGccgBQN",
        "4RIGVSojmy-R",
        "RX5PuSUmvRri",
        "X9uiFJ-ogBRi",
        "u13ffZdjrqQf"
      ]
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "metadata": {
        "id": "JS6jP7tpCrLr",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Digit Recognizer\n",
        "Learn computer vision fundamentals with the famous MNIST dat\n",
        "\n",
        "https://www.kaggle.com/c/digit-recognizer\n",
        "\n",
        "### Competition Description\n",
        "MNIST (\"Modified National Institute of Standards and Technology\") is the de facto “hello world” dataset of computer vision. Since its release in 1999, this classic dataset of handwritten images has served as the basis for benchmarking classification algorithms. As new machine learning techniques emerge, MNIST remains a reliable resource for researchers and learners alike.\n",
        "\n",
        "In this competition, your goal is to correctly identify digits from a dataset of tens of thousands of handwritten images. We’ve curated a set of tutorial-style kernels which cover everything from regression to neural networks. We encourage you to experiment with different algorithms to learn first-hand what works well and how techniques compare.\n",
        "\n",
        "### Practice Skills\n",
        "Computer vision fundamentals including simple neural networks\n",
        "\n",
        "Classification methods such as SVM and K-nearest neighbors\n",
        "\n",
        "#### Acknowledgements \n",
        "More details about the dataset, including algorithms that have been tried on it and their levels of success, can be found at http://yann.lecun.com/exdb/mnist/index.html. The dataset is made available under a Creative Commons Attribution-Share Alike 3.0 license."
      ]
    },
    {
      "metadata": {
        "id": "LgcAAVmXgBPm",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "import pandas as pd\n",
        "import math\n",
        "import numpy as np\n",
        "import matplotlib.pyplot as plt, matplotlib.image as mpimg\n",
        "from sklearn.model_selection import train_test_split\n",
        "import tensorflow as tf\n",
        "%matplotlib inline"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "EITzxKZRgBPy",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "from tensorflow import keras\n",
        "from tensorflow.keras import models\n",
        "from tensorflow.keras import losses,optimizers,metrics\n",
        "from tensorflow.keras import layers"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "4PihLjAggBP1",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Data Preparation"
      ]
    },
    {
      "metadata": {
        "id": "cra90fdEsmsI",
        "colab_type": "code",
        "outputId": "d873fca1-d2d0-40dc-9c09-9cb07b14a591",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "cell_type": "code",
      "source": [
        "from google.colab import drive \n",
        "drive.mount('/content/gdrive')"
      ],
      "execution_count": 57,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Drive already mounted at /content/gdrive; to attempt to forcibly remount, call drive.mount(\"/content/gdrive\", force_remount=True).\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "K58DeQH2gBP2",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "labeled_images = pd.read_csv('gdrive/My Drive/dataML/train.csv')\n",
        "#labeled_images = pd.read_csv('train.csv')\n",
        "images = labeled_images.iloc[:,1:]\n",
        "labels = labeled_images.iloc[:,:1]\n",
        "train_images, test_images,train_labels, test_labels = train_test_split(images, labels, test_size=0.01)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "At9soAW0qOs4",
        "colab_type": "code",
        "outputId": "f4fa52ce-be55-4e43-dccc-e72632b8fe93",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 85
        }
      },
      "cell_type": "code",
      "source": [
        "print(train_images.shape)\n",
        "print(train_labels.shape)\n",
        "print(test_images.shape)\n",
        "print(test_labels.shape)"
      ],
      "execution_count": 39,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "(41580, 784)\n",
            "(41580, 1)\n",
            "(420, 784)\n",
            "(420, 1)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "f87cEn1_xfqI",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Keras"
      ]
    },
    {
      "metadata": {
        "id": "UfkBgDM1gBP5",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "#### convert the data to the right type"
      ]
    },
    {
      "metadata": {
        "id": "-X3Uu-o_gBP6",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "x_train = train_images.values.reshape(train_images.shape[0],28,28,1)\n",
        "x_test = test_images.values.reshape(test_images.shape[0],28,28,1)\n",
        "y_train = train_labels.values\n",
        "y_test = test_labels.values"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "7vboLIlsgBP9",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 282
        },
        "outputId": "b12b1420-306e-4486-a168-7e5da894cb45"
      },
      "cell_type": "code",
      "source": [
        "plt.imshow(x_train[12].squeeze())"
      ],
      "execution_count": 41,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<matplotlib.image.AxesImage at 0x7f65316dec88>"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 41
        },
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAPoAAAD4CAYAAADFJPs2AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAADyhJREFUeJzt3W2MlfWZx/EvT6NCrBQbS0sKBC1X\ndsEoJSZVl5aqXbqEXU2Y2hfGGFBRUtFkqWEa3oDGtdaoaxHGgGshrE3AmCBaIbQuEYkxS1RqNXJZ\nGhmRUUGMCsvyMAP7Ys7MzpnO/T9nzjNz/T6vzn1fc9/nypGf98P/3Oc/5MyZM4jI4Da03g2ISPUp\n6CIBKOgiASjoIgEo6CIBDK/R++jWvkj1DckqlBx0M3sM+D5dIb7H3XeVui8Rqa6STt3N7IfAd939\nSuBW4DcV7UpEKqrUa/RrgU0A7v4e8HUz+1rFuhKRiio16GOBQ72WD+XWiUgDqtRd98ybACJSf6UG\nvZ38I/i3gY/Lb0dEqqHUoG8DmgHM7HtAu7sfqVhXIlJRQ0p9es3MfgX8ADgN/Nzd/5T4c42ji1Rf\n5iV0yUEfIAVdpPoyg66vwIoEoKCLBKCgiwSgoIsEoKCLBKCgiwSgoIsEoKCLBKCgiwSgoIsEoKCL\nBKCgiwSgoIsEoKCLBKCgiwSgoIsEoKCLBKCgiwSgoIsEoKCLBKCgiwSgoIsEoKCLBKCgiwSgoIsE\noKCLBKCgiwSgoIsEoKCLBDC83g1IttOnT+ctDx06NG/diy++mLntRx99lNz3ueeem6wvW7YsWd+y\nZUve8pQpU3j33XfzlqVxlBR0M5sJPAt0/5f9s7svqlRTIlJZ5RzRX3H35op1IiJVo2t0kQCGnDlz\nZsAb5U7dVwF7gTHAcnf/Q2KTgb+JiAzUkMxCiUEfB/wDsBGYBGwHLnH3kxmbKOgl0M04GaDMoJd0\nje7uB4ANucW/mtknwDjgg1L2JyLVVdI1upndZGa/yL0eC3wTOFDJxkSkcko9dT8f+B0wGmii6xr9\npcQmOnXvx1dffZWsL1myJG+5tbWVhQsX9iyvWbMmc9vFixcn9/3QQw8V0WE8p06dStb37NmTrG/a\ntClZnzNnTs/radOm8dZbb+Utl6nip+5HgH8uuR0RqSkNr4kEoKCLBKCgiwSgoIsEoKCLBKDHVKvo\n0KFDyXpLS0uyvm7durzl1tbWvCG1Rx99NHPb22+/vYgO4yk0pLl06dJkvbW1taz3v/jii3teT5s2\njffeey9vuVp0RBcJQEEXCUBBFwlAQRcJQEEXCUBBFwlAQRcJoKTHVEswKB9T7ezsTNYXLFiQrPcd\nJ++r77jqrl27uOKKK3qWX3nllcxtR44cmdz3YHbyZP4PHTU1NfWsmzVrVnLbV199taz3LvR48IMP\nPtjzuu8vBg0dWvZxN/MxVR3RRQJQ0EUCUNBFAlDQRQJQ0EUCUNBFAlDQRQLQOHoZ2trakvXezx73\n5/zzz0/WN2/enLc8Y8aMvHHeGTNmFOhwcOo7Tt7Xvffem7f8+OOPc8899wCwcuXKst573LhxyXqh\nn4M+77zzynr/AjSOLhKZgi4SgIIuEoCCLhKAgi4SgIIuEoCCLhKAxtELSE2jO3ny5OS2+/fvT9YL\njYNv3749WR+sCo2Tv/RSaoZuaG5uzlvu6Ohg+PDipjCYO3dusr5q1apk/cILLyzqfaqkvGmTzWwq\n8DzwmLs/YWbfAdYDw4CPgZvd/UQlOhWRyit46m5mo4AVwMu9Vt8HrHT3GcBeYH512hORSijmGv0E\nMBto77VuJtD9/cwXgOsq25aIVFLBU3d37wA6zKz36lG9TtUPAt+qQm8NYcSIEZm1Dz74oIadxNHU\n1JSs33DDDcl6R0dHUesiqcQki5k3AAYD3YyrPd2Mq7xSh9eOmln3YzjjyD+tF5EGU2rQ/wh0/69v\nLrC1Mu2ISDUUPJ8xs+nAI8BE4JSZNQM3AWvN7A6gDUj/QPlZrPfvbvdV6NS8kMsuu6ys7QerQs+M\n933efCCWLFlS1r5Hjx5d8nvXUzE3496g6y57Xz+ueDciUhX6CqxIAAq6SAAKukgACrpIAAq6SAB6\nTLWA9evXZ9bmzZuX3LbQUMybb76ZrI8fPz5Zb2Tt7dnfofriiy+S2xb6xuCXX36ZrPf9Sea2tjYm\nTJgAFP7MG/WbbUXSzz2LRKagiwSgoIsEoKCLBKCgiwSgoIsEoKCLBKBx9AJSY+FHjx5Nbjt27Nhk\n/e23307Wx4wZk6yX49NPP03Wd+7cmaz3fdxz7969XHLJJT3Lhw8fzty20L+5cj/X1157LW95/Pjx\nfPjhhz2vBzGNo4tEpqCLBKCgiwSgoIsEoKCLBKCgiwSgoIsEUImZWga1I0eOZNaGDElPUvPJJ58k\n68uXL0/W+/4c9Pz583n66aeT23R75513kvXVq1cn68ePHy/qfXrbt29fz+vUWHmhz62QBQsWJOv9\njZUP8vHzgnREFwlAQRcJQEEXCUBBFwlAQRcJQEEXCUBBFwlAz6MXsGrVqszaokWLatgJdHZ2MmzY\nsIrsa9SoUcn6Aw88kKz3/e31yy+/nN27d/csT5s2LXPboUPTx5eWlpayegss8wsKRX1hxsymAs8D\nj7n7E2a2FpgOdP+6wMPu/vtyuxSR6igYdDMbBawAXu5T+qW7v1iVrkSkooq5Rj8BzAay59gRkYZW\n9DW6mS0DPut16j4WaAIOAne5+2eJzc/aa3SRs0h51+j9WA8cdvfdZtYCLAPuKnFfDU034/qnm3Fn\nl5KC7u69r9c3A62VaUdEqqGkcXQze87MJuUWZwLpZyJFpK4KXqOb2XTgEWAicAo4QNdd+BbgGHAU\nmOfuBxO7OWuv0Ts7OzNr8+fPT277zDPPJOtNTU3J+kUXXZS3vG/fPiZOnNizfPfdd2du2/vv+jN7\n9uxkfc+ePcn6VVddlbd87NgxRo4c2bN84sSJzG0LXX5s3749Wb/66quT9cBKv0Z39zfoOmr39VwZ\nDYlIDekrsCIBKOgiASjoIgEo6CIBKOgiAegx1TKcOnUqWd+xY0eynpqSGWD69OkD7qlSJkyYkKwf\nOHAgb7mjo4Phw/9/EOecc87J3Hbbtm3JfWv4rGSaNlkkMgVdJAAFXSQABV0kAAVdJAAFXSQABV0k\nAE2bXIYRI0Yk69dee22NOhm4nTt3Juuff/55Wfu/8847M2saJ689HdFFAlDQRQJQ0EUCUNBFAlDQ\nRQJQ0EUCUNBFAtDz6IPUsWPHkvVLL700WW9ra0vWJ02alLf8/vvvM3ny5J7l119/PXPbCy64ILnv\n48ePJ+uFZpkJTM+ji0SmoIsEoKCLBKCgiwSgoIsEoKCLBKCgiwSg59EHqY0bNybrhcbJC5k1a1Zy\n3f79+zO3XbNmTXLfc+bMSdanTJlSoDvpq6igm9mvgRm5v38Q2AWsB4YBHwM3u3v2hNgiUlcFT93N\n7EfAVHe/EvgJ8O/AfcBKd58B7AXmV7VLESlLMdfoO4Cf5l5/AYwCZgKbc+teAK6reGciUjED+q67\nmS2g6xR+lrtflFt3MbDe3a9KbKrvuotUX+Z33Yu+GWdm1wO3Av8I/KWYnUv9rF27Nlm/7bbbytr/\nwoUL85ZXrFjBokWLitr/1q1bk/vWzbjKK2p4zcxmAUuBf3L3L4GjZnZerjwOaK9SfyJSAQVP3c3s\nAuBV4Dp3P5hbtxrY4e7/aWa/Ad5296cSu9GpexWkjow33nhjcttCj7EOVN9pk1P/rpqbm5P72rBh\nQ8X6CqasU/efAd8ANppZ97pbgKfM7A6gDVhXbociUj0Fg+7uq4HV/ZR+XPl2RKQa9BVYkQAUdJEA\nFHSRABR0kQAUdJEA9JjqWezJJ5/MrFV6nHyg+nuMtdu6dRqNrTUd0UUCUNBFAlDQRQJQ0EUCUNBF\nAlDQRQJQ0EUC0Di6lOSaa65Jrtu0aVPmtk1NTVXpSbLpiC4SgIIuEoCCLhKAgi4SgIIuEoCCLhKA\ngi4SwICmZCqDfte9CrZs2ZJZu+OOO5Lbtren59y4//77k/XFixfnLTc1NXHy5Mm8Zam5zN911xFd\nJAAFXSQABV0kAAVdJAAFXSQABV0kAAVdJICixtHN7NfADLqeX38Q+BdgOnA49ycPu/vvE7vQOLpI\n9ZU+P7qZ/QiY6u5XmtmFwFvAfwG/dPcXK9ejiFRLMb8wswP479zrL4BRwLCqdSQiFTegr8Ca2QK6\nTuE7gbFAE3AQuMvdP0tsqlN3keor/yuwZnY9cCtwF7AeaHH3a4DdwLIyGxSRKirqxyHNbBawFPiJ\nu38JvNyrvBlorUJvIlIhBY/oZnYB8DAwx90/z617zswm5f5kJvBO1ToUkbIVc0T/GfANYKOZda/7\nLbDBzI4BR4F51WlPRCpBz6OLDB56Hl0kMgVdJAAFXSQABV0kAAVdJAAFXSQABV0kAAVdJAAFXSQA\nBV0kAAVdJAAFXSQABV0kAAVdJICifmGmAjIfnxOR6tMRXSQABV0kAAVdJAAFXSQABV0kAAVdJAAF\nXSSAWo2j9zCzx4Dv0/UT0Pe4+65a99AfM5sJPAu8m1v1Z3dfVL+OwMymAs8Dj7n7E2b2HbqmwxoG\nfAzc7O4nGqS3tQxsKu1q9tZ3mu9dNMDnVoHpx0tW06Cb2Q+B7+amYP474Gngylr2UMAr7t5c7yYA\nzGwUsIL86a/uA1a6+7Nm9m/AfOowHVZGb9AAU2lnTPP9MnX+3Oo9/XitT92vBTYBuPt7wNfN7Gs1\n7uFscQKYDbT3WjeTrrnuAF4ArqtxT936661R7AB+mnvdPc33TOr/ufXXV82mH6/1qftY4I1ey4dy\n676qcR9Z/t7MNgNjgOXu/od6NeLuHUBHr2mwAEb1OuU8CHyr5o2R2RvAXWb2rxQ3lXa1eusE/ie3\neCvwEjCr3p9bRl+d1Ogzq/fNuEb6DvxfgOXA9cAtwH+YWVN9W0pqpM8OGmwq7T7TfPdW18+tXtOP\n1/qI3k7XEbzbt+m6OVJ37n4A2JBb/KuZfQKMAz6oX1d/46iZnefu/0tXbw1z6uzuDTOVdt9pvs2s\nIT63ek4/Xusj+jagGcDMvge0u/uRGvfQLzO7ycx+kXs9FvgmcKC+Xf2NPwJzc6/nAlvr2EueRplK\nu79pvmmAz63e04/XajbVHmb2K+AHwGng5+7+p5o2kMHMzgd+B4wGmui6Rn+pjv1MBx4BJgKn6Pqf\nzk3AWuBcoA2Y5+6nGqS3FUAL0DOVtrsfrENvC+g6BX6/1+pbgKeo4+eW0ddv6TqFr/pnVvOgi0jt\n1ftmnIjUgIIuEoCCLhKAgi4SgIIuEoCCLhKAgi4SwP8BB3VSJacShhIAAAAASUVORK5CYII=\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "metadata": {
        "id": "b6I1adl5gBQD",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "#### convert the data to the right type"
      ]
    },
    {
      "metadata": {
        "id": "GAIvNCv6gBQE",
        "colab_type": "code",
        "outputId": "e6f42659-0766-4f2c-df33-fba496026756",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 68
        }
      },
      "cell_type": "code",
      "source": [
        "x_train = train_images\n",
        "x_test = test_images\n",
        "x_train /= 255\n",
        "x_test /= 255\n",
        "y_train = train_labels\n",
        "y_test = test_labels\n",
        "print('x_train shape:', x_train.shape)\n",
        "print(x_train.shape[0], 'train samples')\n",
        "print(x_test.shape[0], 'test samples')"
      ],
      "execution_count": 42,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "x_train shape: (41580, 784)\n",
            "41580 train samples\n",
            "420 test samples\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "collapsed": true,
        "id": "xeMZR7ntgBQI",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "### convert class vectors to binary class matrices - this is for use in the\n",
        "### categorical_crossentropy loss below"
      ]
    },
    {
      "metadata": {
        "id": "8e2qjHPLgBQJ",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "y_train = keras.utils.to_categorical(y_train)\n",
        "y_test = keras.utils.to_categorical(y_test)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "LT78eGccgBQN",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "### Creating the Model\n"
      ]
    },
    {
      "metadata": {
        "id": "AMOStnPCFWSI",
        "colab_type": "code",
        "outputId": "67e94478-d67d-40c4-e1c6-b1f9840d6c3e",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 513
        }
      },
      "cell_type": "code",
      "source": [
        "model = models.Sequential()\n",
        "\n",
        "model.add(layers.Dense(units=200, activation='relu',input_shape=(784,)))\n",
        "model.add(layers.Dropout(0.25))\n",
        "\n",
        "model.add(layers.Dense(units=100, activation='relu'))\n",
        "model.add(layers.Dropout(0.25))\n",
        "\n",
        "model.add(layers.Dense(units=60, activation='relu'))\n",
        "model.add(layers.Dropout(0.25))\n",
        "\n",
        "model.add(layers.Dense(units=30, activation='relu'))\n",
        "model.add(layers.Dropout(0.25))\n",
        "\n",
        "model.add(layers.Dense(units=10, activation='softmax'))\n",
        "\n",
        "model.summary()          "
      ],
      "execution_count": 46,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/layers/core.py:143: calling dropout (from tensorflow.python.ops.nn_ops) with keep_prob is deprecated and will be removed in a future version.\n",
            "Instructions for updating:\n",
            "Please use `rate` instead of `keep_prob`. Rate should be set to `rate = 1 - keep_prob`.\n",
            "_________________________________________________________________\n",
            "Layer (type)                 Output Shape              Param #   \n",
            "=================================================================\n",
            "dense (Dense)                (None, 200)               157000    \n",
            "_________________________________________________________________\n",
            "dropout (Dropout)            (None, 200)               0         \n",
            "_________________________________________________________________\n",
            "dense_1 (Dense)              (None, 100)               20100     \n",
            "_________________________________________________________________\n",
            "dropout_1 (Dropout)          (None, 100)               0         \n",
            "_________________________________________________________________\n",
            "dense_2 (Dense)              (None, 60)                6060      \n",
            "_________________________________________________________________\n",
            "dropout_2 (Dropout)          (None, 60)                0         \n",
            "_________________________________________________________________\n",
            "dense_3 (Dense)              (None, 30)                1830      \n",
            "_________________________________________________________________\n",
            "dropout_3 (Dropout)          (None, 30)                0         \n",
            "_________________________________________________________________\n",
            "dense_4 (Dense)              (None, 10)                310       \n",
            "=================================================================\n",
            "Total params: 185,300\n",
            "Trainable params: 185,300\n",
            "Non-trainable params: 0\n",
            "_________________________________________________________________\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "BLdOV8vYgBQS",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "adam = keras.optimizers.Adam(lr = 0.0001)\n",
        "\n",
        "model.compile(loss=keras.losses.categorical_crossentropy, \n",
        "              optimizer=adam, \n",
        "              metrics=['accuracy'])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "pxI5Gi4xgBQV",
        "colab_type": "code",
        "outputId": "8f81e033-49ed-4cbb-bb7b-0bb92596fc75",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 10234
        }
      },
      "cell_type": "code",
      "source": [
        "H = model.fit(x_train, y_train,\n",
        "          batch_size=100,\n",
        "          epochs=300,\n",
        "          verbose=1,\n",
        "          validation_data=(x_test, y_test))"
      ],
      "execution_count": 54,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Train on 41580 samples, validate on 420 samples\n",
            "Epoch 1/300\n",
            "41580/41580 [==============================] - 5s 118us/sample - loss: 0.7477 - acc: 0.7655 - val_loss: 0.3542 - val_acc: 0.9095\n",
            "Epoch 2/300\n",
            "41580/41580 [==============================] - 4s 106us/sample - loss: 0.5722 - acc: 0.8306 - val_loss: 0.2722 - val_acc: 0.9143\n",
            "Epoch 3/300\n",
            "41580/41580 [==============================] - 4s 106us/sample - loss: 0.4821 - acc: 0.8623 - val_loss: 0.2321 - val_acc: 0.9381\n",
            "Epoch 4/300\n",
            "41580/41580 [==============================] - 4s 107us/sample - loss: 0.4089 - acc: 0.8867 - val_loss: 0.2080 - val_acc: 0.9405\n",
            "Epoch 5/300\n",
            "41580/41580 [==============================] - 4s 107us/sample - loss: 0.3600 - acc: 0.9008 - val_loss: 0.1891 - val_acc: 0.9452\n",
            "Epoch 6/300\n",
            "41580/41580 [==============================] - 4s 107us/sample - loss: 0.3222 - acc: 0.9113 - val_loss: 0.1781 - val_acc: 0.9405\n",
            "Epoch 7/300\n",
            "41580/41580 [==============================] - 4s 107us/sample - loss: 0.2986 - acc: 0.9197 - val_loss: 0.1643 - val_acc: 0.9524\n",
            "Epoch 8/300\n",
            "41580/41580 [==============================] - 4s 108us/sample - loss: 0.2733 - acc: 0.9261 - val_loss: 0.1603 - val_acc: 0.9524\n",
            "Epoch 9/300\n",
            "41580/41580 [==============================] - 4s 107us/sample - loss: 0.2532 - acc: 0.9317 - val_loss: 0.1529 - val_acc: 0.9524\n",
            "Epoch 10/300\n",
            "41580/41580 [==============================] - 4s 107us/sample - loss: 0.2361 - acc: 0.9372 - val_loss: 0.1488 - val_acc: 0.9500\n",
            "Epoch 11/300\n",
            "41580/41580 [==============================] - 4s 106us/sample - loss: 0.2207 - acc: 0.9411 - val_loss: 0.1499 - val_acc: 0.9548\n",
            "Epoch 12/300\n",
            "41580/41580 [==============================] - 4s 106us/sample - loss: 0.2042 - acc: 0.9464 - val_loss: 0.1316 - val_acc: 0.9548\n",
            "Epoch 13/300\n",
            "41580/41580 [==============================] - 4s 107us/sample - loss: 0.1924 - acc: 0.9486 - val_loss: 0.1291 - val_acc: 0.9619\n",
            "Epoch 14/300\n",
            "41580/41580 [==============================] - 4s 106us/sample - loss: 0.1845 - acc: 0.9508 - val_loss: 0.1294 - val_acc: 0.9619\n",
            "Epoch 15/300\n",
            "41580/41580 [==============================] - 4s 107us/sample - loss: 0.1737 - acc: 0.9544 - val_loss: 0.1229 - val_acc: 0.9595\n",
            "Epoch 16/300\n",
            "41580/41580 [==============================] - 4s 107us/sample - loss: 0.1643 - acc: 0.9565 - val_loss: 0.1137 - val_acc: 0.9667\n",
            "Epoch 17/300\n",
            "41580/41580 [==============================] - 4s 106us/sample - loss: 0.1587 - acc: 0.9576 - val_loss: 0.1219 - val_acc: 0.9643\n",
            "Epoch 18/300\n",
            "41580/41580 [==============================] - 4s 106us/sample - loss: 0.1498 - acc: 0.9607 - val_loss: 0.1167 - val_acc: 0.9667\n",
            "Epoch 19/300\n",
            "41580/41580 [==============================] - 4s 107us/sample - loss: 0.1445 - acc: 0.9621 - val_loss: 0.1144 - val_acc: 0.9667\n",
            "Epoch 20/300\n",
            "41580/41580 [==============================] - 4s 107us/sample - loss: 0.1379 - acc: 0.9650 - val_loss: 0.1148 - val_acc: 0.9690\n",
            "Epoch 21/300\n",
            "41580/41580 [==============================] - 4s 106us/sample - loss: 0.1309 - acc: 0.9649 - val_loss: 0.1063 - val_acc: 0.9690\n",
            "Epoch 22/300\n",
            "41580/41580 [==============================] - 4s 107us/sample - loss: 0.1282 - acc: 0.9657 - val_loss: 0.1119 - val_acc: 0.9690\n",
            "Epoch 23/300\n",
            "41580/41580 [==============================] - 4s 106us/sample - loss: 0.1219 - acc: 0.9678 - val_loss: 0.1175 - val_acc: 0.9667\n",
            "Epoch 24/300\n",
            "41580/41580 [==============================] - 4s 106us/sample - loss: 0.1158 - acc: 0.9698 - val_loss: 0.1144 - val_acc: 0.9690\n",
            "Epoch 25/300\n",
            "41580/41580 [==============================] - 4s 106us/sample - loss: 0.1104 - acc: 0.9714 - val_loss: 0.1218 - val_acc: 0.9667\n",
            "Epoch 26/300\n",
            "41580/41580 [==============================] - 4s 107us/sample - loss: 0.1094 - acc: 0.9718 - val_loss: 0.1012 - val_acc: 0.9690\n",
            "Epoch 27/300\n",
            "41580/41580 [==============================] - 4s 106us/sample - loss: 0.1045 - acc: 0.9726 - val_loss: 0.1001 - val_acc: 0.9714\n",
            "Epoch 28/300\n",
            "41580/41580 [==============================] - 4s 106us/sample - loss: 0.1030 - acc: 0.9729 - val_loss: 0.1008 - val_acc: 0.9738\n",
            "Epoch 29/300\n",
            "41580/41580 [==============================] - 4s 106us/sample - loss: 0.0970 - acc: 0.9743 - val_loss: 0.1141 - val_acc: 0.9738\n",
            "Epoch 30/300\n",
            "41580/41580 [==============================] - 4s 107us/sample - loss: 0.0943 - acc: 0.9747 - val_loss: 0.1156 - val_acc: 0.9714\n",
            "Epoch 31/300\n",
            "41580/41580 [==============================] - 4s 106us/sample - loss: 0.0907 - acc: 0.9755 - val_loss: 0.1176 - val_acc: 0.9714\n",
            "Epoch 32/300\n",
            "41580/41580 [==============================] - 4s 106us/sample - loss: 0.0867 - acc: 0.9774 - val_loss: 0.1128 - val_acc: 0.9762\n",
            "Epoch 33/300\n",
            "41580/41580 [==============================] - 4s 107us/sample - loss: 0.0853 - acc: 0.9778 - val_loss: 0.1214 - val_acc: 0.9714\n",
            "Epoch 34/300\n",
            "41580/41580 [==============================] - 4s 107us/sample - loss: 0.0802 - acc: 0.9788 - val_loss: 0.0992 - val_acc: 0.9738\n",
            "Epoch 35/300\n",
            "41580/41580 [==============================] - 4s 107us/sample - loss: 0.0785 - acc: 0.9794 - val_loss: 0.1172 - val_acc: 0.9690\n",
            "Epoch 36/300\n",
            "41580/41580 [==============================] - 4s 107us/sample - loss: 0.0779 - acc: 0.9789 - val_loss: 0.1168 - val_acc: 0.9714\n",
            "Epoch 37/300\n",
            "41580/41580 [==============================] - 4s 106us/sample - loss: 0.0739 - acc: 0.9800 - val_loss: 0.1139 - val_acc: 0.9714\n",
            "Epoch 38/300\n",
            "41580/41580 [==============================] - 4s 106us/sample - loss: 0.0711 - acc: 0.9804 - val_loss: 0.1130 - val_acc: 0.9714\n",
            "Epoch 39/300\n",
            "41580/41580 [==============================] - 4s 107us/sample - loss: 0.0688 - acc: 0.9819 - val_loss: 0.1053 - val_acc: 0.9738\n",
            "Epoch 40/300\n",
            "41580/41580 [==============================] - 4s 106us/sample - loss: 0.0689 - acc: 0.9808 - val_loss: 0.1233 - val_acc: 0.9762\n",
            "Epoch 41/300\n",
            "41580/41580 [==============================] - 4s 106us/sample - loss: 0.0663 - acc: 0.9822 - val_loss: 0.1180 - val_acc: 0.9738\n",
            "Epoch 42/300\n",
            "41580/41580 [==============================] - 4s 107us/sample - loss: 0.0608 - acc: 0.9832 - val_loss: 0.1244 - val_acc: 0.9714\n",
            "Epoch 43/300\n",
            "41580/41580 [==============================] - 4s 106us/sample - loss: 0.0607 - acc: 0.9844 - val_loss: 0.1212 - val_acc: 0.9738\n",
            "Epoch 44/300\n",
            "41580/41580 [==============================] - 4s 106us/sample - loss: 0.0576 - acc: 0.9844 - val_loss: 0.1090 - val_acc: 0.9762\n",
            "Epoch 45/300\n",
            "41580/41580 [==============================] - 4s 106us/sample - loss: 0.0614 - acc: 0.9841 - val_loss: 0.1098 - val_acc: 0.9738\n",
            "Epoch 46/300\n",
            "41580/41580 [==============================] - 4s 106us/sample - loss: 0.0587 - acc: 0.9837 - val_loss: 0.1202 - val_acc: 0.9738\n",
            "Epoch 47/300\n",
            "41580/41580 [==============================] - 4s 106us/sample - loss: 0.0558 - acc: 0.9850 - val_loss: 0.1333 - val_acc: 0.9714\n",
            "Epoch 48/300\n",
            "41580/41580 [==============================] - 4s 107us/sample - loss: 0.0560 - acc: 0.9844 - val_loss: 0.1296 - val_acc: 0.9714\n",
            "Epoch 49/300\n",
            "41580/41580 [==============================] - 4s 108us/sample - loss: 0.0523 - acc: 0.9854 - val_loss: 0.1357 - val_acc: 0.9738\n",
            "Epoch 50/300\n",
            "41580/41580 [==============================] - 4s 107us/sample - loss: 0.0540 - acc: 0.9856 - val_loss: 0.1286 - val_acc: 0.9738\n",
            "Epoch 51/300\n",
            "41580/41580 [==============================] - 4s 105us/sample - loss: 0.0499 - acc: 0.9860 - val_loss: 0.1165 - val_acc: 0.9738\n",
            "Epoch 52/300\n",
            "41580/41580 [==============================] - 4s 106us/sample - loss: 0.0480 - acc: 0.9868 - val_loss: 0.1333 - val_acc: 0.9738\n",
            "Epoch 53/300\n",
            "41580/41580 [==============================] - 4s 105us/sample - loss: 0.0493 - acc: 0.9865 - val_loss: 0.1328 - val_acc: 0.9714\n",
            "Epoch 54/300\n",
            "41580/41580 [==============================] - 4s 107us/sample - loss: 0.0483 - acc: 0.9867 - val_loss: 0.1210 - val_acc: 0.9738\n",
            "Epoch 55/300\n",
            "41580/41580 [==============================] - 4s 106us/sample - loss: 0.0444 - acc: 0.9878 - val_loss: 0.1374 - val_acc: 0.9738\n",
            "Epoch 56/300\n",
            "41580/41580 [==============================] - 4s 107us/sample - loss: 0.0434 - acc: 0.9881 - val_loss: 0.1324 - val_acc: 0.9762\n",
            "Epoch 57/300\n",
            "41580/41580 [==============================] - 4s 105us/sample - loss: 0.0452 - acc: 0.9877 - val_loss: 0.1369 - val_acc: 0.9738\n",
            "Epoch 58/300\n",
            "41580/41580 [==============================] - 4s 106us/sample - loss: 0.0435 - acc: 0.9886 - val_loss: 0.1337 - val_acc: 0.9738\n",
            "Epoch 59/300\n",
            "41580/41580 [==============================] - 4s 107us/sample - loss: 0.0420 - acc: 0.9888 - val_loss: 0.1259 - val_acc: 0.9786\n",
            "Epoch 60/300\n",
            "41580/41580 [==============================] - 4s 107us/sample - loss: 0.0406 - acc: 0.9883 - val_loss: 0.1329 - val_acc: 0.9762\n",
            "Epoch 61/300\n",
            "41580/41580 [==============================] - 4s 106us/sample - loss: 0.0414 - acc: 0.9889 - val_loss: 0.1327 - val_acc: 0.9714\n",
            "Epoch 62/300\n",
            "41580/41580 [==============================] - 4s 106us/sample - loss: 0.0391 - acc: 0.9892 - val_loss: 0.1314 - val_acc: 0.9738\n",
            "Epoch 63/300\n",
            "41580/41580 [==============================] - 4s 105us/sample - loss: 0.0370 - acc: 0.9902 - val_loss: 0.1269 - val_acc: 0.9762\n",
            "Epoch 64/300\n",
            "41580/41580 [==============================] - 4s 105us/sample - loss: 0.0348 - acc: 0.9905 - val_loss: 0.1458 - val_acc: 0.9714\n",
            "Epoch 65/300\n",
            "41580/41580 [==============================] - 4s 106us/sample - loss: 0.0392 - acc: 0.9892 - val_loss: 0.1432 - val_acc: 0.9762\n",
            "Epoch 66/300\n",
            "41580/41580 [==============================] - 4s 106us/sample - loss: 0.0374 - acc: 0.9898 - val_loss: 0.1553 - val_acc: 0.9714\n",
            "Epoch 67/300\n",
            "41580/41580 [==============================] - 4s 106us/sample - loss: 0.0358 - acc: 0.9908 - val_loss: 0.1456 - val_acc: 0.9738\n",
            "Epoch 68/300\n",
            "41580/41580 [==============================] - 4s 107us/sample - loss: 0.0339 - acc: 0.9911 - val_loss: 0.1415 - val_acc: 0.9762\n",
            "Epoch 69/300\n",
            "41580/41580 [==============================] - 4s 106us/sample - loss: 0.0349 - acc: 0.9911 - val_loss: 0.1700 - val_acc: 0.9738\n",
            "Epoch 70/300\n",
            "41580/41580 [==============================] - 4s 106us/sample - loss: 0.0332 - acc: 0.9910 - val_loss: 0.1500 - val_acc: 0.9762\n",
            "Epoch 71/300\n",
            "41580/41580 [==============================] - 4s 106us/sample - loss: 0.0331 - acc: 0.9912 - val_loss: 0.1388 - val_acc: 0.9738\n",
            "Epoch 72/300\n",
            "41580/41580 [==============================] - 4s 107us/sample - loss: 0.0305 - acc: 0.9917 - val_loss: 0.1209 - val_acc: 0.9738\n",
            "Epoch 73/300\n",
            "41580/41580 [==============================] - 4s 107us/sample - loss: 0.0340 - acc: 0.9910 - val_loss: 0.1141 - val_acc: 0.9738\n",
            "Epoch 74/300\n",
            "41580/41580 [==============================] - 4s 106us/sample - loss: 0.0324 - acc: 0.9919 - val_loss: 0.1432 - val_acc: 0.9762\n",
            "Epoch 75/300\n",
            "41580/41580 [==============================] - 4s 106us/sample - loss: 0.0302 - acc: 0.9920 - val_loss: 0.1581 - val_acc: 0.9738\n",
            "Epoch 76/300\n",
            "41580/41580 [==============================] - 4s 107us/sample - loss: 0.0268 - acc: 0.9929 - val_loss: 0.1241 - val_acc: 0.9738\n",
            "Epoch 77/300\n",
            "41580/41580 [==============================] - 4s 106us/sample - loss: 0.0294 - acc: 0.9917 - val_loss: 0.1399 - val_acc: 0.9786\n",
            "Epoch 78/300\n",
            "41580/41580 [==============================] - 4s 108us/sample - loss: 0.0293 - acc: 0.9922 - val_loss: 0.1225 - val_acc: 0.9762\n",
            "Epoch 79/300\n",
            "41580/41580 [==============================] - 4s 106us/sample - loss: 0.0274 - acc: 0.9930 - val_loss: 0.1527 - val_acc: 0.9762\n",
            "Epoch 80/300\n",
            "41580/41580 [==============================] - 4s 107us/sample - loss: 0.0290 - acc: 0.9920 - val_loss: 0.1298 - val_acc: 0.9786\n",
            "Epoch 81/300\n",
            "41580/41580 [==============================] - 4s 105us/sample - loss: 0.0276 - acc: 0.9922 - val_loss: 0.1287 - val_acc: 0.9762\n",
            "Epoch 82/300\n",
            "41580/41580 [==============================] - 4s 106us/sample - loss: 0.0259 - acc: 0.9928 - val_loss: 0.1454 - val_acc: 0.9738\n",
            "Epoch 83/300\n",
            "41580/41580 [==============================] - 4s 105us/sample - loss: 0.0279 - acc: 0.9924 - val_loss: 0.1612 - val_acc: 0.9738\n",
            "Epoch 84/300\n",
            "41580/41580 [==============================] - 4s 105us/sample - loss: 0.0250 - acc: 0.9937 - val_loss: 0.1641 - val_acc: 0.9714\n",
            "Epoch 85/300\n",
            "41580/41580 [==============================] - 4s 106us/sample - loss: 0.0258 - acc: 0.9929 - val_loss: 0.1769 - val_acc: 0.9738\n",
            "Epoch 86/300\n",
            "41580/41580 [==============================] - 4s 106us/sample - loss: 0.0274 - acc: 0.9925 - val_loss: 0.1626 - val_acc: 0.9762\n",
            "Epoch 87/300\n",
            "41580/41580 [==============================] - 4s 106us/sample - loss: 0.0248 - acc: 0.9939 - val_loss: 0.1502 - val_acc: 0.9762\n",
            "Epoch 88/300\n",
            "41580/41580 [==============================] - 4s 105us/sample - loss: 0.0242 - acc: 0.9935 - val_loss: 0.1823 - val_acc: 0.9714\n",
            "Epoch 89/300\n",
            "41580/41580 [==============================] - 4s 105us/sample - loss: 0.0260 - acc: 0.9932 - val_loss: 0.1528 - val_acc: 0.9762\n",
            "Epoch 90/300\n",
            "41580/41580 [==============================] - 4s 106us/sample - loss: 0.0233 - acc: 0.9941 - val_loss: 0.1427 - val_acc: 0.9786\n",
            "Epoch 91/300\n",
            "41580/41580 [==============================] - 4s 106us/sample - loss: 0.0254 - acc: 0.9931 - val_loss: 0.1535 - val_acc: 0.9738\n",
            "Epoch 92/300\n",
            "41580/41580 [==============================] - 4s 106us/sample - loss: 0.0241 - acc: 0.9938 - val_loss: 0.1298 - val_acc: 0.9762\n",
            "Epoch 93/300\n",
            "41580/41580 [==============================] - 4s 107us/sample - loss: 0.0255 - acc: 0.9932 - val_loss: 0.1823 - val_acc: 0.9738\n",
            "Epoch 94/300\n",
            "41580/41580 [==============================] - 4s 106us/sample - loss: 0.0247 - acc: 0.9936 - val_loss: 0.1604 - val_acc: 0.9738\n",
            "Epoch 95/300\n",
            "41580/41580 [==============================] - 4s 106us/sample - loss: 0.0222 - acc: 0.9940 - val_loss: 0.1611 - val_acc: 0.9738\n",
            "Epoch 96/300\n",
            "41580/41580 [==============================] - 4s 106us/sample - loss: 0.0236 - acc: 0.9938 - val_loss: 0.1751 - val_acc: 0.9738\n",
            "Epoch 97/300\n",
            "41580/41580 [==============================] - 4s 106us/sample - loss: 0.0209 - acc: 0.9945 - val_loss: 0.1550 - val_acc: 0.9762\n",
            "Epoch 98/300\n",
            "41580/41580 [==============================] - 4s 105us/sample - loss: 0.0217 - acc: 0.9942 - val_loss: 0.1538 - val_acc: 0.9762\n",
            "Epoch 99/300\n",
            "41580/41580 [==============================] - 4s 106us/sample - loss: 0.0205 - acc: 0.9945 - val_loss: 0.1587 - val_acc: 0.9762\n",
            "Epoch 100/300\n",
            "41580/41580 [==============================] - 4s 106us/sample - loss: 0.0208 - acc: 0.9948 - val_loss: 0.1408 - val_acc: 0.9762\n",
            "Epoch 101/300\n",
            "41580/41580 [==============================] - 4s 107us/sample - loss: 0.0210 - acc: 0.9943 - val_loss: 0.1553 - val_acc: 0.9738\n",
            "Epoch 102/300\n",
            "41580/41580 [==============================] - 4s 105us/sample - loss: 0.0211 - acc: 0.9943 - val_loss: 0.1296 - val_acc: 0.9762\n",
            "Epoch 103/300\n",
            "41580/41580 [==============================] - 4s 106us/sample - loss: 0.0207 - acc: 0.9947 - val_loss: 0.1349 - val_acc: 0.9762\n",
            "Epoch 104/300\n",
            "41580/41580 [==============================] - 4s 105us/sample - loss: 0.0228 - acc: 0.9938 - val_loss: 0.1174 - val_acc: 0.9762\n",
            "Epoch 105/300\n",
            "41580/41580 [==============================] - 4s 106us/sample - loss: 0.0190 - acc: 0.9949 - val_loss: 0.1303 - val_acc: 0.9810\n",
            "Epoch 106/300\n",
            "41580/41580 [==============================] - 4s 105us/sample - loss: 0.0222 - acc: 0.9943 - val_loss: 0.1509 - val_acc: 0.9714\n",
            "Epoch 107/300\n",
            "41580/41580 [==============================] - 4s 106us/sample - loss: 0.0201 - acc: 0.9944 - val_loss: 0.1356 - val_acc: 0.9786\n",
            "Epoch 108/300\n",
            "41580/41580 [==============================] - 4s 107us/sample - loss: 0.0197 - acc: 0.9948 - val_loss: 0.1312 - val_acc: 0.9714\n",
            "Epoch 109/300\n",
            "41580/41580 [==============================] - 4s 106us/sample - loss: 0.0195 - acc: 0.9946 - val_loss: 0.1252 - val_acc: 0.9738\n",
            "Epoch 110/300\n",
            "41580/41580 [==============================] - 4s 107us/sample - loss: 0.0198 - acc: 0.9944 - val_loss: 0.1581 - val_acc: 0.9738\n",
            "Epoch 111/300\n",
            "41580/41580 [==============================] - 4s 107us/sample - loss: 0.0218 - acc: 0.9943 - val_loss: 0.1436 - val_acc: 0.9762\n",
            "Epoch 112/300\n",
            "41580/41580 [==============================] - 4s 106us/sample - loss: 0.0193 - acc: 0.9949 - val_loss: 0.1551 - val_acc: 0.9786\n",
            "Epoch 113/300\n",
            "41580/41580 [==============================] - 4s 106us/sample - loss: 0.0166 - acc: 0.9958 - val_loss: 0.1531 - val_acc: 0.9690\n",
            "Epoch 114/300\n",
            "41580/41580 [==============================] - 4s 106us/sample - loss: 0.0195 - acc: 0.9949 - val_loss: 0.1617 - val_acc: 0.9714\n",
            "Epoch 115/300\n",
            "41580/41580 [==============================] - 4s 107us/sample - loss: 0.0181 - acc: 0.9953 - val_loss: 0.1789 - val_acc: 0.9690\n",
            "Epoch 116/300\n",
            "41580/41580 [==============================] - 4s 106us/sample - loss: 0.0186 - acc: 0.9948 - val_loss: 0.1611 - val_acc: 0.9738\n",
            "Epoch 117/300\n",
            "41580/41580 [==============================] - 4s 106us/sample - loss: 0.0192 - acc: 0.9948 - val_loss: 0.1520 - val_acc: 0.9762\n",
            "Epoch 118/300\n",
            "41580/41580 [==============================] - 4s 106us/sample - loss: 0.0174 - acc: 0.9956 - val_loss: 0.1568 - val_acc: 0.9786\n",
            "Epoch 119/300\n",
            "41580/41580 [==============================] - 4s 107us/sample - loss: 0.0179 - acc: 0.9953 - val_loss: 0.1594 - val_acc: 0.9786\n",
            "Epoch 120/300\n",
            "41580/41580 [==============================] - 4s 106us/sample - loss: 0.0165 - acc: 0.9960 - val_loss: 0.1427 - val_acc: 0.9762\n",
            "Epoch 121/300\n",
            "41580/41580 [==============================] - 4s 106us/sample - loss: 0.0177 - acc: 0.9951 - val_loss: 0.1540 - val_acc: 0.9786\n",
            "Epoch 122/300\n",
            "41580/41580 [==============================] - 4s 106us/sample - loss: 0.0184 - acc: 0.9951 - val_loss: 0.1555 - val_acc: 0.9786\n",
            "Epoch 123/300\n",
            "41580/41580 [==============================] - 4s 105us/sample - loss: 0.0165 - acc: 0.9958 - val_loss: 0.1608 - val_acc: 0.9762\n",
            "Epoch 124/300\n",
            "41580/41580 [==============================] - 4s 106us/sample - loss: 0.0169 - acc: 0.9957 - val_loss: 0.1672 - val_acc: 0.9738\n",
            "Epoch 125/300\n",
            "41580/41580 [==============================] - 4s 106us/sample - loss: 0.0164 - acc: 0.9957 - val_loss: 0.1473 - val_acc: 0.9738\n",
            "Epoch 126/300\n",
            "41580/41580 [==============================] - 4s 106us/sample - loss: 0.0185 - acc: 0.9951 - val_loss: 0.1547 - val_acc: 0.9786\n",
            "Epoch 127/300\n",
            "41580/41580 [==============================] - 4s 106us/sample - loss: 0.0178 - acc: 0.9956 - val_loss: 0.1573 - val_acc: 0.9762\n",
            "Epoch 128/300\n",
            "41580/41580 [==============================] - 4s 106us/sample - loss: 0.0180 - acc: 0.9952 - val_loss: 0.1427 - val_acc: 0.9786\n",
            "Epoch 129/300\n",
            "41580/41580 [==============================] - 4s 108us/sample - loss: 0.0168 - acc: 0.9958 - val_loss: 0.1781 - val_acc: 0.9762\n",
            "Epoch 130/300\n",
            "41580/41580 [==============================] - 4s 107us/sample - loss: 0.0160 - acc: 0.9958 - val_loss: 0.1408 - val_acc: 0.9786\n",
            "Epoch 131/300\n",
            "41580/41580 [==============================] - 4s 107us/sample - loss: 0.0184 - acc: 0.9953 - val_loss: 0.1463 - val_acc: 0.9762\n",
            "Epoch 132/300\n",
            "41580/41580 [==============================] - 4s 107us/sample - loss: 0.0169 - acc: 0.9961 - val_loss: 0.1643 - val_acc: 0.9786\n",
            "Epoch 133/300\n",
            "41580/41580 [==============================] - 4s 105us/sample - loss: 0.0150 - acc: 0.9961 - val_loss: 0.1714 - val_acc: 0.9810\n",
            "Epoch 134/300\n",
            "41580/41580 [==============================] - 4s 105us/sample - loss: 0.0169 - acc: 0.9959 - val_loss: 0.1484 - val_acc: 0.9762\n",
            "Epoch 135/300\n",
            "41580/41580 [==============================] - 4s 106us/sample - loss: 0.0168 - acc: 0.9962 - val_loss: 0.1505 - val_acc: 0.9786\n",
            "Epoch 136/300\n",
            "41580/41580 [==============================] - 4s 107us/sample - loss: 0.0150 - acc: 0.9965 - val_loss: 0.1497 - val_acc: 0.9810\n",
            "Epoch 137/300\n",
            "41580/41580 [==============================] - 4s 106us/sample - loss: 0.0144 - acc: 0.9961 - val_loss: 0.1587 - val_acc: 0.9762\n",
            "Epoch 138/300\n",
            "41580/41580 [==============================] - 4s 106us/sample - loss: 0.0154 - acc: 0.9958 - val_loss: 0.1357 - val_acc: 0.9833\n",
            "Epoch 139/300\n",
            "41580/41580 [==============================] - 4s 107us/sample - loss: 0.0158 - acc: 0.9960 - val_loss: 0.1496 - val_acc: 0.9738\n",
            "Epoch 140/300\n",
            "41580/41580 [==============================] - 4s 106us/sample - loss: 0.0141 - acc: 0.9965 - val_loss: 0.1599 - val_acc: 0.9738\n",
            "Epoch 141/300\n",
            "41580/41580 [==============================] - 4s 107us/sample - loss: 0.0145 - acc: 0.9962 - val_loss: 0.1595 - val_acc: 0.9786\n",
            "Epoch 142/300\n",
            "41580/41580 [==============================] - 4s 106us/sample - loss: 0.0163 - acc: 0.9958 - val_loss: 0.1629 - val_acc: 0.9786\n",
            "Epoch 143/300\n",
            "41580/41580 [==============================] - 4s 106us/sample - loss: 0.0147 - acc: 0.9962 - val_loss: 0.1747 - val_acc: 0.9738\n",
            "Epoch 144/300\n",
            "41580/41580 [==============================] - 4s 106us/sample - loss: 0.0129 - acc: 0.9966 - val_loss: 0.1659 - val_acc: 0.9738\n",
            "Epoch 145/300\n",
            "41580/41580 [==============================] - 4s 106us/sample - loss: 0.0148 - acc: 0.9958 - val_loss: 0.1511 - val_acc: 0.9738\n",
            "Epoch 146/300\n",
            "41580/41580 [==============================] - 4s 106us/sample - loss: 0.0136 - acc: 0.9961 - val_loss: 0.1696 - val_acc: 0.9738\n",
            "Epoch 147/300\n",
            "41580/41580 [==============================] - 4s 106us/sample - loss: 0.0138 - acc: 0.9963 - val_loss: 0.1423 - val_acc: 0.9810\n",
            "Epoch 148/300\n",
            "41580/41580 [==============================] - 4s 106us/sample - loss: 0.0130 - acc: 0.9967 - val_loss: 0.1515 - val_acc: 0.9738\n",
            "Epoch 149/300\n",
            "41580/41580 [==============================] - 4s 102us/sample - loss: 0.0144 - acc: 0.9962 - val_loss: 0.1411 - val_acc: 0.9810\n",
            "Epoch 150/300\n",
            "41580/41580 [==============================] - 4s 106us/sample - loss: 0.0136 - acc: 0.9963 - val_loss: 0.1545 - val_acc: 0.9810\n",
            "Epoch 151/300\n",
            "41580/41580 [==============================] - 4s 107us/sample - loss: 0.0152 - acc: 0.9959 - val_loss: 0.1567 - val_acc: 0.9762\n",
            "Epoch 152/300\n",
            "41580/41580 [==============================] - 4s 106us/sample - loss: 0.0140 - acc: 0.9963 - val_loss: 0.1683 - val_acc: 0.9786\n",
            "Epoch 153/300\n",
            "41580/41580 [==============================] - 4s 106us/sample - loss: 0.0141 - acc: 0.9967 - val_loss: 0.1660 - val_acc: 0.9738\n",
            "Epoch 154/300\n",
            "41580/41580 [==============================] - 4s 108us/sample - loss: 0.0138 - acc: 0.9965 - val_loss: 0.1656 - val_acc: 0.9786\n",
            "Epoch 155/300\n",
            "41580/41580 [==============================] - 4s 107us/sample - loss: 0.0127 - acc: 0.9968 - val_loss: 0.1928 - val_acc: 0.9786\n",
            "Epoch 156/300\n",
            "41580/41580 [==============================] - 4s 106us/sample - loss: 0.0126 - acc: 0.9966 - val_loss: 0.1471 - val_acc: 0.9762\n",
            "Epoch 157/300\n",
            "41580/41580 [==============================] - 4s 106us/sample - loss: 0.0137 - acc: 0.9965 - val_loss: 0.1447 - val_acc: 0.9762\n",
            "Epoch 158/300\n",
            "41580/41580 [==============================] - 4s 106us/sample - loss: 0.0124 - acc: 0.9967 - val_loss: 0.1895 - val_acc: 0.9762\n",
            "Epoch 159/300\n",
            "41580/41580 [==============================] - 4s 107us/sample - loss: 0.0118 - acc: 0.9968 - val_loss: 0.1888 - val_acc: 0.9738\n",
            "Epoch 160/300\n",
            "41580/41580 [==============================] - 4s 107us/sample - loss: 0.0119 - acc: 0.9968 - val_loss: 0.1911 - val_acc: 0.9738\n",
            "Epoch 161/300\n",
            "41580/41580 [==============================] - 4s 106us/sample - loss: 0.0133 - acc: 0.9966 - val_loss: 0.1540 - val_acc: 0.9762\n",
            "Epoch 162/300\n",
            "41580/41580 [==============================] - 4s 106us/sample - loss: 0.0129 - acc: 0.9969 - val_loss: 0.1252 - val_acc: 0.9786\n",
            "Epoch 163/300\n",
            "41580/41580 [==============================] - 4s 106us/sample - loss: 0.0144 - acc: 0.9962 - val_loss: 0.1547 - val_acc: 0.9762\n",
            "Epoch 164/300\n",
            "41580/41580 [==============================] - 4s 107us/sample - loss: 0.0117 - acc: 0.9972 - val_loss: 0.1449 - val_acc: 0.9786\n",
            "Epoch 165/300\n",
            "41580/41580 [==============================] - 4s 106us/sample - loss: 0.0141 - acc: 0.9967 - val_loss: 0.1454 - val_acc: 0.9786\n",
            "Epoch 166/300\n",
            "41580/41580 [==============================] - 4s 106us/sample - loss: 0.0118 - acc: 0.9968 - val_loss: 0.1341 - val_acc: 0.9810\n",
            "Epoch 167/300\n",
            "41580/41580 [==============================] - 4s 105us/sample - loss: 0.0139 - acc: 0.9964 - val_loss: 0.1228 - val_acc: 0.9833\n",
            "Epoch 168/300\n",
            "41580/41580 [==============================] - 4s 107us/sample - loss: 0.0110 - acc: 0.9974 - val_loss: 0.1155 - val_acc: 0.9810\n",
            "Epoch 169/300\n",
            "41580/41580 [==============================] - 4s 107us/sample - loss: 0.0107 - acc: 0.9971 - val_loss: 0.1415 - val_acc: 0.9762\n",
            "Epoch 170/300\n",
            "41580/41580 [==============================] - 4s 106us/sample - loss: 0.0125 - acc: 0.9970 - val_loss: 0.1384 - val_acc: 0.9762\n",
            "Epoch 171/300\n",
            "41580/41580 [==============================] - 4s 107us/sample - loss: 0.0117 - acc: 0.9966 - val_loss: 0.1353 - val_acc: 0.9810\n",
            "Epoch 172/300\n",
            "41580/41580 [==============================] - 4s 107us/sample - loss: 0.0127 - acc: 0.9965 - val_loss: 0.1385 - val_acc: 0.9786\n",
            "Epoch 173/300\n",
            "41580/41580 [==============================] - 4s 106us/sample - loss: 0.0128 - acc: 0.9965 - val_loss: 0.1391 - val_acc: 0.9810\n",
            "Epoch 174/300\n",
            "41580/41580 [==============================] - 4s 107us/sample - loss: 0.0126 - acc: 0.9968 - val_loss: 0.1138 - val_acc: 0.9786\n",
            "Epoch 175/300\n",
            "41580/41580 [==============================] - 4s 106us/sample - loss: 0.0109 - acc: 0.9973 - val_loss: 0.1261 - val_acc: 0.9786\n",
            "Epoch 176/300\n",
            "41580/41580 [==============================] - 4s 108us/sample - loss: 0.0114 - acc: 0.9971 - val_loss: 0.1683 - val_acc: 0.9786\n",
            "Epoch 177/300\n",
            "41580/41580 [==============================] - 4s 107us/sample - loss: 0.0121 - acc: 0.9969 - val_loss: 0.1309 - val_acc: 0.9762\n",
            "Epoch 178/300\n",
            "41580/41580 [==============================] - 4s 107us/sample - loss: 0.0117 - acc: 0.9972 - val_loss: 0.1645 - val_acc: 0.9762\n",
            "Epoch 179/300\n",
            "41580/41580 [==============================] - 4s 106us/sample - loss: 0.0123 - acc: 0.9969 - val_loss: 0.1168 - val_acc: 0.9810\n",
            "Epoch 180/300\n",
            "41580/41580 [==============================] - 4s 106us/sample - loss: 0.0127 - acc: 0.9967 - val_loss: 0.1530 - val_acc: 0.9738\n",
            "Epoch 181/300\n",
            "41580/41580 [==============================] - 4s 106us/sample - loss: 0.0113 - acc: 0.9974 - val_loss: 0.1514 - val_acc: 0.9786\n",
            "Epoch 182/300\n",
            "41580/41580 [==============================] - 4s 106us/sample - loss: 0.0123 - acc: 0.9969 - val_loss: 0.1757 - val_acc: 0.9762\n",
            "Epoch 183/300\n",
            "41580/41580 [==============================] - 4s 106us/sample - loss: 0.0124 - acc: 0.9969 - val_loss: 0.1588 - val_acc: 0.9762\n",
            "Epoch 184/300\n",
            "41580/41580 [==============================] - 4s 106us/sample - loss: 0.0113 - acc: 0.9972 - val_loss: 0.1663 - val_acc: 0.9786\n",
            "Epoch 185/300\n",
            "41580/41580 [==============================] - 4s 107us/sample - loss: 0.0140 - acc: 0.9962 - val_loss: 0.1528 - val_acc: 0.9786\n",
            "Epoch 186/300\n",
            "41580/41580 [==============================] - 4s 105us/sample - loss: 0.0127 - acc: 0.9968 - val_loss: 0.1368 - val_acc: 0.9810\n",
            "Epoch 187/300\n",
            "41580/41580 [==============================] - 4s 104us/sample - loss: 0.0089 - acc: 0.9978 - val_loss: 0.1546 - val_acc: 0.9762\n",
            "Epoch 188/300\n",
            "41580/41580 [==============================] - 4s 107us/sample - loss: 0.0115 - acc: 0.9970 - val_loss: 0.1839 - val_acc: 0.9786\n",
            "Epoch 189/300\n",
            "41580/41580 [==============================] - 4s 107us/sample - loss: 0.0133 - acc: 0.9968 - val_loss: 0.1875 - val_acc: 0.9786\n",
            "Epoch 190/300\n",
            "41580/41580 [==============================] - 4s 106us/sample - loss: 0.0105 - acc: 0.9971 - val_loss: 0.1728 - val_acc: 0.9786\n",
            "Epoch 191/300\n",
            "41580/41580 [==============================] - 4s 106us/sample - loss: 0.0101 - acc: 0.9975 - val_loss: 0.1600 - val_acc: 0.9786\n",
            "Epoch 192/300\n",
            "41580/41580 [==============================] - 4s 106us/sample - loss: 0.0124 - acc: 0.9972 - val_loss: 0.1548 - val_acc: 0.9762\n",
            "Epoch 193/300\n",
            "41580/41580 [==============================] - 4s 106us/sample - loss: 0.0115 - acc: 0.9975 - val_loss: 0.1849 - val_acc: 0.9762\n",
            "Epoch 194/300\n",
            "41580/41580 [==============================] - 4s 106us/sample - loss: 0.0113 - acc: 0.9971 - val_loss: 0.1824 - val_acc: 0.9762\n",
            "Epoch 195/300\n",
            "41580/41580 [==============================] - 4s 107us/sample - loss: 0.0096 - acc: 0.9975 - val_loss: 0.1741 - val_acc: 0.9738\n",
            "Epoch 196/300\n",
            "41580/41580 [==============================] - 4s 106us/sample - loss: 0.0103 - acc: 0.9974 - val_loss: 0.1645 - val_acc: 0.9738\n",
            "Epoch 197/300\n",
            "41580/41580 [==============================] - 4s 106us/sample - loss: 0.0122 - acc: 0.9969 - val_loss: 0.1400 - val_acc: 0.9810\n",
            "Epoch 198/300\n",
            "41580/41580 [==============================] - 4s 106us/sample - loss: 0.0104 - acc: 0.9975 - val_loss: 0.1778 - val_acc: 0.9786\n",
            "Epoch 199/300\n",
            "41580/41580 [==============================] - 4s 107us/sample - loss: 0.0108 - acc: 0.9972 - val_loss: 0.1495 - val_acc: 0.9762\n",
            "Epoch 200/300\n",
            "41580/41580 [==============================] - 4s 107us/sample - loss: 0.0108 - acc: 0.9972 - val_loss: 0.1625 - val_acc: 0.9786\n",
            "Epoch 201/300\n",
            "41580/41580 [==============================] - 4s 107us/sample - loss: 0.0107 - acc: 0.9971 - val_loss: 0.1481 - val_acc: 0.9786\n",
            "Epoch 202/300\n",
            "41580/41580 [==============================] - 4s 106us/sample - loss: 0.0111 - acc: 0.9975 - val_loss: 0.1862 - val_acc: 0.9762\n",
            "Epoch 203/300\n",
            "41580/41580 [==============================] - 4s 105us/sample - loss: 0.0111 - acc: 0.9974 - val_loss: 0.1538 - val_acc: 0.9786\n",
            "Epoch 204/300\n",
            "41580/41580 [==============================] - 4s 105us/sample - loss: 0.0095 - acc: 0.9975 - val_loss: 0.1503 - val_acc: 0.9786\n",
            "Epoch 205/300\n",
            "41580/41580 [==============================] - 4s 104us/sample - loss: 0.0111 - acc: 0.9969 - val_loss: 0.1424 - val_acc: 0.9810\n",
            "Epoch 206/300\n",
            "41580/41580 [==============================] - 4s 106us/sample - loss: 0.0087 - acc: 0.9975 - val_loss: 0.1564 - val_acc: 0.9714\n",
            "Epoch 207/300\n",
            "41580/41580 [==============================] - 4s 107us/sample - loss: 0.0108 - acc: 0.9974 - val_loss: 0.1717 - val_acc: 0.9762\n",
            "Epoch 208/300\n",
            "41580/41580 [==============================] - 4s 106us/sample - loss: 0.0117 - acc: 0.9970 - val_loss: 0.1503 - val_acc: 0.9786\n",
            "Epoch 209/300\n",
            "41580/41580 [==============================] - 4s 107us/sample - loss: 0.0116 - acc: 0.9969 - val_loss: 0.1500 - val_acc: 0.9762\n",
            "Epoch 210/300\n",
            "41580/41580 [==============================] - 4s 106us/sample - loss: 0.0103 - acc: 0.9974 - val_loss: 0.1250 - val_acc: 0.9833\n",
            "Epoch 211/300\n",
            "41580/41580 [==============================] - 4s 106us/sample - loss: 0.0105 - acc: 0.9970 - val_loss: 0.1336 - val_acc: 0.9810\n",
            "Epoch 212/300\n",
            "41580/41580 [==============================] - 4s 106us/sample - loss: 0.0100 - acc: 0.9975 - val_loss: 0.1373 - val_acc: 0.9810\n",
            "Epoch 213/300\n",
            "41580/41580 [==============================] - 4s 106us/sample - loss: 0.0098 - acc: 0.9978 - val_loss: 0.1073 - val_acc: 0.9857\n",
            "Epoch 214/300\n",
            "41580/41580 [==============================] - 4s 106us/sample - loss: 0.0120 - acc: 0.9970 - val_loss: 0.1300 - val_acc: 0.9810\n",
            "Epoch 215/300\n",
            "41580/41580 [==============================] - 4s 106us/sample - loss: 0.0104 - acc: 0.9977 - val_loss: 0.1117 - val_acc: 0.9833\n",
            "Epoch 216/300\n",
            "41580/41580 [==============================] - 4s 106us/sample - loss: 0.0098 - acc: 0.9973 - val_loss: 0.1392 - val_acc: 0.9786\n",
            "Epoch 217/300\n",
            "41580/41580 [==============================] - 4s 106us/sample - loss: 0.0085 - acc: 0.9978 - val_loss: 0.1216 - val_acc: 0.9810\n",
            "Epoch 218/300\n",
            "41580/41580 [==============================] - 4s 106us/sample - loss: 0.0103 - acc: 0.9977 - val_loss: 0.1422 - val_acc: 0.9786\n",
            "Epoch 219/300\n",
            "41580/41580 [==============================] - 4s 106us/sample - loss: 0.0098 - acc: 0.9972 - val_loss: 0.1468 - val_acc: 0.9762\n",
            "Epoch 220/300\n",
            "41580/41580 [==============================] - 4s 107us/sample - loss: 0.0114 - acc: 0.9972 - val_loss: 0.1699 - val_acc: 0.9762\n",
            "Epoch 221/300\n",
            "41580/41580 [==============================] - 4s 107us/sample - loss: 0.0095 - acc: 0.9978 - val_loss: 0.1716 - val_acc: 0.9786\n",
            "Epoch 222/300\n",
            "41580/41580 [==============================] - 4s 107us/sample - loss: 0.0102 - acc: 0.9974 - val_loss: 0.1769 - val_acc: 0.9714\n",
            "Epoch 223/300\n",
            "41580/41580 [==============================] - 4s 106us/sample - loss: 0.0084 - acc: 0.9977 - val_loss: 0.1792 - val_acc: 0.9786\n",
            "Epoch 224/300\n",
            "41580/41580 [==============================] - 4s 107us/sample - loss: 0.0090 - acc: 0.9974 - val_loss: 0.1681 - val_acc: 0.9738\n",
            "Epoch 225/300\n",
            "41580/41580 [==============================] - 4s 106us/sample - loss: 0.0093 - acc: 0.9974 - val_loss: 0.1599 - val_acc: 0.9762\n",
            "Epoch 226/300\n",
            "41580/41580 [==============================] - 4s 107us/sample - loss: 0.0091 - acc: 0.9978 - val_loss: 0.1445 - val_acc: 0.9786\n",
            "Epoch 227/300\n",
            "41580/41580 [==============================] - 4s 107us/sample - loss: 0.0106 - acc: 0.9975 - val_loss: 0.1631 - val_acc: 0.9762\n",
            "Epoch 228/300\n",
            "41580/41580 [==============================] - 4s 106us/sample - loss: 0.0091 - acc: 0.9978 - val_loss: 0.1608 - val_acc: 0.9762\n",
            "Epoch 229/300\n",
            "41580/41580 [==============================] - 4s 106us/sample - loss: 0.0094 - acc: 0.9975 - val_loss: 0.1822 - val_acc: 0.9762\n",
            "Epoch 230/300\n",
            "41580/41580 [==============================] - 4s 106us/sample - loss: 0.0087 - acc: 0.9977 - val_loss: 0.1600 - val_acc: 0.9810\n",
            "Epoch 231/300\n",
            "41580/41580 [==============================] - 4s 107us/sample - loss: 0.0092 - acc: 0.9973 - val_loss: 0.1695 - val_acc: 0.9786\n",
            "Epoch 232/300\n",
            "41580/41580 [==============================] - 4s 106us/sample - loss: 0.0075 - acc: 0.9978 - val_loss: 0.1813 - val_acc: 0.9786\n",
            "Epoch 233/300\n",
            "41580/41580 [==============================] - 4s 106us/sample - loss: 0.0102 - acc: 0.9974 - val_loss: 0.1704 - val_acc: 0.9786\n",
            "Epoch 234/300\n",
            "41580/41580 [==============================] - 4s 108us/sample - loss: 0.0101 - acc: 0.9976 - val_loss: 0.1641 - val_acc: 0.9810\n",
            "Epoch 235/300\n",
            "41580/41580 [==============================] - 4s 106us/sample - loss: 0.0094 - acc: 0.9975 - val_loss: 0.1648 - val_acc: 0.9786\n",
            "Epoch 236/300\n",
            "41580/41580 [==============================] - 4s 107us/sample - loss: 0.0084 - acc: 0.9980 - val_loss: 0.1723 - val_acc: 0.9738\n",
            "Epoch 237/300\n",
            "41580/41580 [==============================] - 4s 107us/sample - loss: 0.0119 - acc: 0.9974 - val_loss: 0.1722 - val_acc: 0.9786\n",
            "Epoch 238/300\n",
            "41580/41580 [==============================] - 4s 107us/sample - loss: 0.0100 - acc: 0.9976 - val_loss: 0.1556 - val_acc: 0.9762\n",
            "Epoch 239/300\n",
            "41580/41580 [==============================] - 4s 106us/sample - loss: 0.0081 - acc: 0.9977 - val_loss: 0.1594 - val_acc: 0.9810\n",
            "Epoch 240/300\n",
            "41580/41580 [==============================] - 4s 106us/sample - loss: 0.0084 - acc: 0.9978 - val_loss: 0.1584 - val_acc: 0.9762\n",
            "Epoch 241/300\n",
            "41580/41580 [==============================] - 4s 106us/sample - loss: 0.0079 - acc: 0.9981 - val_loss: 0.1954 - val_acc: 0.9786\n",
            "Epoch 242/300\n",
            "41580/41580 [==============================] - 4s 106us/sample - loss: 0.0087 - acc: 0.9981 - val_loss: 0.1880 - val_acc: 0.9738\n",
            "Epoch 243/300\n",
            "41580/41580 [==============================] - 4s 106us/sample - loss: 0.0070 - acc: 0.9982 - val_loss: 0.1857 - val_acc: 0.9738\n",
            "Epoch 244/300\n",
            "41580/41580 [==============================] - 4s 106us/sample - loss: 0.0094 - acc: 0.9977 - val_loss: 0.1867 - val_acc: 0.9762\n",
            "Epoch 245/300\n",
            "41580/41580 [==============================] - 4s 107us/sample - loss: 0.0098 - acc: 0.9974 - val_loss: 0.1580 - val_acc: 0.9762\n",
            "Epoch 246/300\n",
            "41580/41580 [==============================] - 4s 107us/sample - loss: 0.0095 - acc: 0.9978 - val_loss: 0.2101 - val_acc: 0.9714\n",
            "Epoch 247/300\n",
            "41580/41580 [==============================] - 4s 105us/sample - loss: 0.0085 - acc: 0.9980 - val_loss: 0.1808 - val_acc: 0.9738\n",
            "Epoch 248/300\n",
            "41580/41580 [==============================] - 4s 107us/sample - loss: 0.0088 - acc: 0.9979 - val_loss: 0.2015 - val_acc: 0.9738\n",
            "Epoch 249/300\n",
            "41580/41580 [==============================] - 4s 106us/sample - loss: 0.0087 - acc: 0.9978 - val_loss: 0.1601 - val_acc: 0.9738\n",
            "Epoch 250/300\n",
            "41580/41580 [==============================] - 4s 106us/sample - loss: 0.0085 - acc: 0.9979 - val_loss: 0.1823 - val_acc: 0.9738\n",
            "Epoch 251/300\n",
            "41580/41580 [==============================] - 4s 107us/sample - loss: 0.0075 - acc: 0.9981 - val_loss: 0.1789 - val_acc: 0.9762\n",
            "Epoch 252/300\n",
            "41580/41580 [==============================] - 4s 107us/sample - loss: 0.0076 - acc: 0.9982 - val_loss: 0.1820 - val_acc: 0.9762\n",
            "Epoch 253/300\n",
            "41580/41580 [==============================] - 4s 106us/sample - loss: 0.0102 - acc: 0.9976 - val_loss: 0.1504 - val_acc: 0.9738\n",
            "Epoch 254/300\n",
            "41580/41580 [==============================] - 4s 105us/sample - loss: 0.0092 - acc: 0.9977 - val_loss: 0.1830 - val_acc: 0.9738\n",
            "Epoch 255/300\n",
            "41580/41580 [==============================] - 4s 106us/sample - loss: 0.0099 - acc: 0.9975 - val_loss: 0.1768 - val_acc: 0.9738\n",
            "Epoch 256/300\n",
            "41580/41580 [==============================] - 4s 106us/sample - loss: 0.0093 - acc: 0.9976 - val_loss: 0.1779 - val_acc: 0.9810\n",
            "Epoch 257/300\n",
            "41580/41580 [==============================] - 4s 106us/sample - loss: 0.0096 - acc: 0.9977 - val_loss: 0.1504 - val_acc: 0.9810\n",
            "Epoch 258/300\n",
            "41580/41580 [==============================] - 4s 105us/sample - loss: 0.0072 - acc: 0.9982 - val_loss: 0.1766 - val_acc: 0.9762\n",
            "Epoch 259/300\n",
            "41580/41580 [==============================] - 4s 105us/sample - loss: 0.0093 - acc: 0.9980 - val_loss: 0.1898 - val_acc: 0.9690\n",
            "Epoch 260/300\n",
            "41580/41580 [==============================] - 4s 105us/sample - loss: 0.0074 - acc: 0.9982 - val_loss: 0.1731 - val_acc: 0.9690\n",
            "Epoch 261/300\n",
            "41580/41580 [==============================] - 4s 106us/sample - loss: 0.0112 - acc: 0.9974 - val_loss: 0.1660 - val_acc: 0.9762\n",
            "Epoch 262/300\n",
            "41580/41580 [==============================] - 4s 106us/sample - loss: 0.0094 - acc: 0.9975 - val_loss: 0.1945 - val_acc: 0.9667\n",
            "Epoch 263/300\n",
            "41580/41580 [==============================] - 4s 106us/sample - loss: 0.0087 - acc: 0.9977 - val_loss: 0.1582 - val_acc: 0.9714\n",
            "Epoch 264/300\n",
            "41580/41580 [==============================] - 4s 107us/sample - loss: 0.0064 - acc: 0.9984 - val_loss: 0.1677 - val_acc: 0.9762\n",
            "Epoch 265/300\n",
            "41580/41580 [==============================] - 4s 106us/sample - loss: 0.0083 - acc: 0.9977 - val_loss: 0.1689 - val_acc: 0.9762\n",
            "Epoch 266/300\n",
            "41580/41580 [==============================] - 4s 107us/sample - loss: 0.0087 - acc: 0.9979 - val_loss: 0.1662 - val_acc: 0.9762\n",
            "Epoch 267/300\n",
            "41580/41580 [==============================] - 4s 106us/sample - loss: 0.0080 - acc: 0.9980 - val_loss: 0.1630 - val_acc: 0.9762\n",
            "Epoch 268/300\n",
            "41580/41580 [==============================] - 4s 107us/sample - loss: 0.0084 - acc: 0.9982 - val_loss: 0.1723 - val_acc: 0.9762\n",
            "Epoch 269/300\n",
            "41580/41580 [==============================] - 4s 106us/sample - loss: 0.0080 - acc: 0.9980 - val_loss: 0.1850 - val_acc: 0.9762\n",
            "Epoch 270/300\n",
            "41580/41580 [==============================] - 4s 106us/sample - loss: 0.0086 - acc: 0.9979 - val_loss: 0.1770 - val_acc: 0.9786\n",
            "Epoch 271/300\n",
            "41580/41580 [==============================] - 4s 107us/sample - loss: 0.0078 - acc: 0.9982 - val_loss: 0.1561 - val_acc: 0.9786\n",
            "Epoch 272/300\n",
            "41580/41580 [==============================] - 4s 106us/sample - loss: 0.0074 - acc: 0.9981 - val_loss: 0.1467 - val_acc: 0.9810\n",
            "Epoch 273/300\n",
            "41580/41580 [==============================] - 4s 105us/sample - loss: 0.0116 - acc: 0.9975 - val_loss: 0.1446 - val_acc: 0.9833\n",
            "Epoch 274/300\n",
            "41580/41580 [==============================] - 4s 105us/sample - loss: 0.0081 - acc: 0.9977 - val_loss: 0.1573 - val_acc: 0.9833\n",
            "Epoch 275/300\n",
            "41580/41580 [==============================] - 4s 104us/sample - loss: 0.0085 - acc: 0.9977 - val_loss: 0.1554 - val_acc: 0.9738\n",
            "Epoch 276/300\n",
            "41580/41580 [==============================] - 4s 106us/sample - loss: 0.0089 - acc: 0.9979 - val_loss: 0.1565 - val_acc: 0.9810\n",
            "Epoch 277/300\n",
            "41580/41580 [==============================] - 4s 106us/sample - loss: 0.0075 - acc: 0.9979 - val_loss: 0.1596 - val_acc: 0.9786\n",
            "Epoch 278/300\n",
            "41580/41580 [==============================] - 4s 107us/sample - loss: 0.0082 - acc: 0.9981 - val_loss: 0.1748 - val_acc: 0.9714\n",
            "Epoch 279/300\n",
            "41580/41580 [==============================] - 4s 107us/sample - loss: 0.0090 - acc: 0.9983 - val_loss: 0.1471 - val_acc: 0.9786\n",
            "Epoch 280/300\n",
            "41580/41580 [==============================] - 4s 106us/sample - loss: 0.0069 - acc: 0.9983 - val_loss: 0.1740 - val_acc: 0.9738\n",
            "Epoch 281/300\n",
            "41580/41580 [==============================] - 4s 107us/sample - loss: 0.0079 - acc: 0.9979 - val_loss: 0.1985 - val_acc: 0.9714\n",
            "Epoch 282/300\n",
            "41580/41580 [==============================] - 4s 105us/sample - loss: 0.0077 - acc: 0.9981 - val_loss: 0.1672 - val_acc: 0.9738\n",
            "Epoch 283/300\n",
            "41580/41580 [==============================] - 4s 106us/sample - loss: 0.0076 - acc: 0.9981 - val_loss: 0.1846 - val_acc: 0.9714\n",
            "Epoch 284/300\n",
            "41580/41580 [==============================] - 4s 107us/sample - loss: 0.0087 - acc: 0.9979 - val_loss: 0.1781 - val_acc: 0.9690\n",
            "Epoch 285/300\n",
            "41580/41580 [==============================] - 4s 107us/sample - loss: 0.0079 - acc: 0.9978 - val_loss: 0.1454 - val_acc: 0.9786\n",
            "Epoch 286/300\n",
            "41580/41580 [==============================] - 4s 106us/sample - loss: 0.0085 - acc: 0.9981 - val_loss: 0.1718 - val_acc: 0.9738\n",
            "Epoch 287/300\n",
            "41580/41580 [==============================] - 4s 106us/sample - loss: 0.0081 - acc: 0.9981 - val_loss: 0.1398 - val_acc: 0.9786\n",
            "Epoch 288/300\n",
            "41580/41580 [==============================] - 4s 107us/sample - loss: 0.0069 - acc: 0.9983 - val_loss: 0.1221 - val_acc: 0.9833\n",
            "Epoch 289/300\n",
            "41580/41580 [==============================] - 4s 107us/sample - loss: 0.0089 - acc: 0.9974 - val_loss: 0.1347 - val_acc: 0.9810\n",
            "Epoch 290/300\n",
            "41580/41580 [==============================] - 4s 107us/sample - loss: 0.0082 - acc: 0.9979 - val_loss: 0.1332 - val_acc: 0.9786\n",
            "Epoch 291/300\n",
            "41580/41580 [==============================] - 5s 108us/sample - loss: 0.0086 - acc: 0.9978 - val_loss: 0.1364 - val_acc: 0.9833\n",
            "Epoch 292/300\n",
            "41580/41580 [==============================] - 4s 107us/sample - loss: 0.0071 - acc: 0.9982 - val_loss: 0.1439 - val_acc: 0.9810\n",
            "Epoch 293/300\n",
            "41580/41580 [==============================] - 4s 106us/sample - loss: 0.0079 - acc: 0.9981 - val_loss: 0.1446 - val_acc: 0.9810\n",
            "Epoch 294/300\n",
            "41580/41580 [==============================] - 4s 107us/sample - loss: 0.0086 - acc: 0.9980 - val_loss: 0.1717 - val_acc: 0.9786\n",
            "Epoch 295/300\n",
            "41580/41580 [==============================] - 4s 106us/sample - loss: 0.0060 - acc: 0.9986 - val_loss: 0.1398 - val_acc: 0.9786\n",
            "Epoch 296/300\n",
            "41580/41580 [==============================] - 4s 106us/sample - loss: 0.0074 - acc: 0.9984 - val_loss: 0.1604 - val_acc: 0.9786\n",
            "Epoch 297/300\n",
            "41580/41580 [==============================] - 4s 107us/sample - loss: 0.0061 - acc: 0.9987 - val_loss: 0.1556 - val_acc: 0.9714\n",
            "Epoch 298/300\n",
            "41580/41580 [==============================] - 4s 106us/sample - loss: 0.0076 - acc: 0.9985 - val_loss: 0.1738 - val_acc: 0.9762\n",
            "Epoch 299/300\n",
            "41580/41580 [==============================] - 4s 106us/sample - loss: 0.0083 - acc: 0.9980 - val_loss: 0.1667 - val_acc: 0.9738\n",
            "Epoch 300/300\n",
            "41580/41580 [==============================] - 4s 106us/sample - loss: 0.0090 - acc: 0.9979 - val_loss: 0.1512 - val_acc: 0.9762\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "CHDGYz_Mki4G",
        "colab_type": "code",
        "outputId": "1b6c61da-3632-4606-e1e0-3c829e1c58a9",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "cell_type": "code",
      "source": [
        "H.history.keys()"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "dict_keys(['loss', 'acc', 'val_loss', 'val_acc'])"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 14
        }
      ]
    },
    {
      "metadata": {
        "id": "G1s_nCBWk-91",
        "colab_type": "code",
        "outputId": "b4917a58-50f3-4500-8851-9ba90db6cf34",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 282
        }
      },
      "cell_type": "code",
      "source": [
        "plt.plot(H.history['acc'])\n",
        "plt.plot(H.history['val_acc'],'r')"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[<matplotlib.lines.Line2D at 0x7fd9762b8ba8>]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 112
        },
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXcAAAD4CAYAAAAXUaZHAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJzs3Xl4lNXZ+PHv7JNkskLCvoOHTUQQ\nBQERte77rsWtttpXbW19a1/b2lZrrW2t2rr1Z13rgrjUpdYNtS5sKqCiIBz2PYRA1slk9uf3xzMz\nmUkmyRASMZP7c125SJ5tzpmE+zlzn+WxGIaBEEKI7GI90AUQQgjR+SS4CyFEFpLgLoQQWUiCuxBC\nZCEJ7kIIkYXsB7oAcZWV9R0etlNcnEt1ta8zi9Mt9MR698Q6Q8+st9Q5M6Wl+ZZ027Oi5W632w50\nEQ6Inljvnlhn6Jn1ljrvn6wI7kIIIVJJcBdCiCwkwV0IIbKQBHchhMhCEtyFECILSXAXQogsJMFd\nCCGy0LdmEpMQQnSVxkCYLzfsZcuueo4Y24chffP3+Ro79zTQGAxTmOukIM+J09H6mPRwJMrGnXXU\nNQQZ3Def0kI3FosFb2OI9dtr2V3tIz/PSbHHRZ+SXIrzXftTvbQkuAshvnUaA2F8/lCbxxiGQfK0\ndgtgsVhS9q/eUs17y7fz1cYqwpEoAG8v3cpxkwdx5sxhRKIGX6zbw+ot1QwozePw0WX0LspJXKPe\nF+TT1btZ+GU5WyrqU16/INdB31559O+VS0Gek0jUIGoYVFQ1snpLFY2BSOJYT46D/FwH5Xtbzj61\nWiz87srD6d87bx/eofZJcBdCtGtrRT1fb65mWL98RgwoxG5rO6MbCkfZXe1j/Y5a1m6rZfOuOgpy\nnQzpm8+QvvnkuuxEDYNoFPLcdkoK3RR7XKzfUctHK3ayXO8G4JARvTny4L6U5LtZuWkvqzZVUb7X\nRzAcIRCMEk162JDTYWXskBIOHtGLglwnb326hQ076gAY0DuPw0aX0bckl5cXbOSdZdtYvLKcxkAk\n5RovfrCBYf0KsNks7K7yUeczbzBWi4WJI3vTpySHuoYgtQ1BKmsaWbethrXbalrUv7TIzdSxfeld\n6GZLRT0bd9axt87P6MFFHDSoiAGlHry+INXeANGoeXxnyyi4K6XuAaYCBnC91npp0r4zgJuBADBP\na31/0r4cYCVwm9b6iU4stxBiHxiGQXV9gHAkSnG+C0fSNHfDMPA2hqiqC7C3zk80ajCsXwElBS4a\n/GFe/mgjH3yxg3gMdDqsjOhfSK9CN4V5TnLdduobQlR7A1TX+ams9VNTH0hpVbudNnbt9aHTBMJ0\n+vXKxemwsXxtJcvXVqbsKyvKoTCWFrHbmlrqNd4gX6zfwxfr9yS2HTqqN6ceOZRh/QoS2yYd1JvX\nl2zh3WXbGdYvn0kHlTJuWAmbd9Xz6eoKVm+pxoKF3oVuBvfNZ8yQYo4c15dCT8vUSTAUoaK6kYbG\nEFarBavVQkGek7Kk1v+B0m5wV0rNAkZpracppcYAjwHTYvuswP3AJGAv8KZS6hWt9fbY6TcDVV1S\nciG+heKBsqK6kRynjQGlnpT9jYEwVXV+ehfm4HK2v46IYRhU1vrZssfH2k172VvnJ8dlpyDPSX6O\ng2A4gtcXwusP4XLYKMhzUpDrxB+MUFXnZ0+dn52VDWyv9NLgDyeum+e243TY8AfD+AMR0q3aV5zv\nIhiK0OAP069XLsdPGcSOygZWb6lm9ZbqtOW1ACUFLtTgInoX5TCsbz6jBhXRv3cegWCEbbu9bK2o\nJxSOYrVaEnnovbX+2PviZuYh/Rk1sJDS0nyWryxnyapd+AJhxg4pZuywEgpyna2+X5U1jXy5YS+V\nNY0cOb4vg/u0zK077DbOnDmcM2cOT9k+uE8+Rx3Sn8ZAGIfd2u6nEwCnw8agMk+7xx0ImbTcjwVe\nAdBar1ZKFSulCrTWdUBvoEZrXQmglHoPOA54Qik1GhgLvN41RRfimxM1DHZXN7Jtt5cdlV6K811M\nHFVKYZ6TSDTKivV7+fCLnWzYUYsv0BREB/fxMHNCf0ryXXz8dQVfrN9DKGzmfovzXRR5XITCUYLh\nCBZgWL8CRg0qoqw4h1WbqvhMV7K7pnG/ym4BykpyGTO0BKfdSnV9gBpvgFA4Su/CHNxOG54cB70K\n3JQUuDEMgw0761i/o5aoAefPHslxhw1MCXaNgXAiPeHzh8nPdVCc76Igz9lqUMxx2TlokJmWyKjc\nFksijZOp0qIcjp08MOPj08lxZUe2OpNa9AWWJ/1cGdtWF/s+Xyk1CtgMzAY+iB13F3AdcFkmBSku\nzt2vFdFKS/e99zsb9MR6d6TOFVU+lnxVTllxDsP6F9KnJBerNXWl1C/XV/Lie+uw2az0KnRT5HGx\np7aRLbvq2VZRTyAYSTn+ybc1o4eUsLvax95aPwADSj0cXOahX+88yvc0sHR1Bc+8szZxzoBSD2OH\nlVBR5WPnnga27fbiclhxOe0EQxE+/rqCj7+uSBzvdtqYdnA/RgwopH9vD2UlOfgDEaq9AWq9AdxO\ns7WeH2utV9f5qfEGyHHZKSvOpbQ4h7698joUsAzDwDBo8T59k+Tvu+M6cotK/Ka11oZS6jLMVE0t\nsAmwKKUuBZZorTcppTK66P6s21xamk9lZX37B2aZnljv5nWuqvMz77/r2bCjltJCN2UluQwq83DE\n2D6Jj+/L1uzm8TfX0JjUos5x2Tl0VG+mjuvD4LJ8XvxgAwu/Kk/7mnabhb4leQwq8zCozMPA0jx2\n7mngs7WVrNlchctpY/akAcyeOICBzT6iXzh7BEtWVdDgDzFZlTKkT37KiI5khmFQvtfHuu01VFQ1\nctCgIsYOLcbpsKXWO9dB/+L0HXBDeue22Oata8Tb+lv6rSV/35mfk47FMNp+RoZS6hagXGv9UOzn\njcAhWusWJVBK3QGsAM4ChgMRYCBmZ+vVWut3W3ud/XlYR0/8I4Dsr3cgGGHhV+Vs2+3F5w/R4A9T\nXOhmcKmHUQMLWbuthlcWbCIQipCf68DbGEp0+tltFg5TZTjsVhZ8WY7TYeXMGcMxDINtu72s217D\n3roAYLZWDGBwmYfLThpN35JcqusD1DYEKfI4KSvOwWZNn2po8Iew26y42hjz3Bmy/XedjtQ543PS\nthYyabnPB24FHlJKTQJ2Jgd2pdSbmKmXBuA04C6t9byk/bcAm9sK7KJnMAyD2oYgnhxHSl62xhvg\nyw17sWDmTIsLXCxbs5u3P92Gt7HlWOfFSd97chxc/J1RzDi4H+GIwZ7aRlZurOKDL3Yk0hsDS/P4\n4RnjU8YRG4bB+h21LFlVwYYdtUwd14fjpwxKBPEclz2jccd5bkfH3gwhuli7wV1rvVgptVwptRiI\nAtcqpS4HarXWLwMPY94ADOAOrfWe1q8mskXUMKj3hQgEw5QUuFOCtc8fZleVj1qv2fqtqg+wZVc9\nm8rr8DaGcNqtDOtXwJC++Wwur2Pd9tq0ozVyXHZOnz6Uw8f0wZPjINdtx+p08MmKHazdXoPLYeOU\naUPIj6VfHHYL/Xrl0a9XHscdNhC9tYbtlV6OOqR/i9mEFouFUQOLGDUws849IbqbdtMy3xRJy+y7\nrqr3pvI6Xl24iQZ/iO+dPIZ+vcwWbDRq8MbHW/hoxU6q6wNEouavzGa1UFacQ5HHRUW1j6pYuqO5\n3oVuBpV5qKxpZEdlAwZmSmTUwEImqzJcTht7ahvZU+tnQO88Zh86kFx3avtDftc9h9Q543M6nJYR\nWWRvrZ95/11HOBxl2vi+TBzZG6fDhs8fZvOuOt5bvp3P1zV9+PrdP5dxxUmjOWhQEQ+/9jWrt1ST\n67IzpG8+xR4XToeN3TU+yvf4KN/ro8jjZNywEvr1yqUk35zkUuhxMrDMkzI+2ecPs213PX1KcilK\nMzlECLF/JLhnGZ8/zJ7aRmq8AWq9QcqKcxLTxVes38Mj//k6MZllxYa95Ljs5Oc4UsZSjxxQyFkz\nh1HfGOLxN9fw/15dhdtpwx+McOio3lxx8hg8Oam5ZsMwCIWjbS6mlCzXbUcNLu68igshUkhwzxLb\nd3t585OtfLq6IpEuiXM7bQztm8+arTXYbVYuPVExamARS1bu4uOvd9HgDzF2aDFD+uQzdlgJY4cU\nJ4brDSrz8ODLK6mo9nHxcaM4dvLAtEP5LBZLxoFdCNH1JOfeTRmGQWMUlnyxgy/W72HVJnOVh369\nchk9pJhijzlbcGtFPSs3VbG7upGy4hyuOXN82inZbQlHojQGwomOywOpJ/6uoWfWW+qc8TmSc+8O\n1m2v4fN1e/A2hvD5w0QiUQaWeRjer4DSohw27KxNrO1R72saJnjQwEJOnDqECSN6YU3Tsq7xBloM\nQcyU3Wb9VgR2IUTmJLh/S6zbXsOrCzfx9eaWCzKt2LC3xbYij5OjJw9kaJmH0YOLKCtuOTMx9Xjp\ntBSiJ5HgfoBU1flZrivZVF7HxvI6dlebHZrjhhZzwhGDKS3MSQwD3FrhjR3jY2jfAsYOLaZvSS5l\nZQU97mOrECIzEtwPgJWb9vLQq6sSo1ZyXXYmjuzNSVMHp51UM25YCeOGlXzTxRRCdGMS3LuQYRhs\nLK+joTFE31559C5wM3/pNl74YD02q4ULjhnJxJG9KSvOaXUxKSGE6AgJ7l2gqs7Pwq/KWbxyVyLd\nAuZMzkjUoMjj5NqzD2ZE/8IDWEohRDaT4N7JFq8s58m3NcFQFKfdytRxfehbksuuKnMWZ3G+i0tP\nVNLBKYToUhLcO0kwFOGZd9ay4Mty3E4bl5ygmDq2T9Y81UUI0b1I5NlPPn+YJat28e6ybVRUNzK4\nj4f/OXM8fdoZmiiEEF1JgnsH1XgD/HvRZhavLCcYimKzWjhu8kDOmz0i5cnyQghxIEhw30ehcJR3\nl23j34s3EwhG6F3oZtbE/syc0J+CPJnFKYT4dpDgnqEab4DFK3fxwec72FPrx5Pj4IITRnLUIf0P\n6AOEhRAiHQnubfA2hvhi3R6W6d2s3FhF1DBw2K0cN3kgZ8wcJo9YE0J8a0lwbyYaNfhsbSXvf74D\nvbWGaGzVzGH98pkxoT9HjCkjV4K6EOJbToJ7TDgS5cMvdjJ/6VYqa/wADO9fwKSDSpl0UCl9S2T0\nixCi+5Dgjvmw58feWM3Hqypw2K3Mmtif46cMSjw7VAghupueHdwjEazbt/Hipggfr6pgxIACfnTO\nhJRnfXY31u3biPbtB/ae/asVoqfb9yc3ZBH3k4/Ta8oEov/4B32Kc/hxNw/stq++pGTKBPL++PsD\nXRQhxAHWo4N78NXXALjq/Yf5xThLt3/aUM5Tj2OJRHA//QQEgwe6OEKIA6jHBvegz0/uso/xuvKw\nRyMM/ckPsNS0fApSt9HYiOulFwGwVlXhfPvNA1wgIcSB1GOD+7Jn3yIn2MjmmSfh++mN2LZuIf9H\nP4Ro9JsvjN+P/YvPsH+2DPtny7Bu2rjPl3C98RrWuloCJ54CgPvZp1o91lJZmXgt+2fL2r+p+f1Y\nKiv3uUz7wlJZCX5/5id4vd37ZixEF8uo100pdQ8wFTCA67XWS5P2nQHcDASAeVrr+2Pb/wzMjL3G\nHVrrlzq57B1WUe2j7o13AOh7zkn4zjwbx9JPcb39Js633iB48qnfaHk8/3cDOc8+nfjZsFqpfn8x\nkTFjM76Ge655fsMtt2HdvQvnf9/FWr6TaL/+qQdGoxTPPhLb7orEpvCYsVR/sARaeWCI51f/h+vl\nF6la8hlGnz77ULPMWCt2UTx1EoGzzsF7930ZnVP4/UuxrdVULfsKrD22jSJEq9r9X6GUmgWM0lpP\nA64E7k3aZwXuB04GjgJOU0oNVErNBsbHzjkR+GtXFL4jDMPgmXfWMn7Ll+aGo44Gmw3f//4cAMfS\nT77R8lhqa3C//CKR/gPwXXs9/nMvwBKN4n7mnxlfw7p1C84FHxCceiSR4SPxX3QJlmgU1/PPtjjW\ntnEDtt0VhMZPwHft9YQOnYR99dfYP2293o6FH2L11uN+8bkO1bE9rueexdrgxfXWGxCbNNYmrxfH\nRx9g274N2+Z9/5QjRE+QSZPnWOAVAK31aqBYKVUQ29cbqNFaV2qto8B7wHHAR8B5sWNqgDyl1Ldi\nqcTP1+1hzdoKxpWvITx6LEZpKQDh8QcDYP/qy2+0PK6X/4XF76fxe1fR8NvbqP/bg0R7l5qBNBDI\n6Bruec8A4L/4EgACZ52D4XbjnvtUi2Bp/2qFecz5F9Lw29touPlW8xqtpHEs9XXYY2ki97Mtr7ff\nDCPx2tY9ldj0mnZPcXy6BEvYfP7sN/37EqK7yCQt0xdYnvRzZWxbXez7fKXUKGAzMBv4QGsdARpi\nx18JvBHb1qri4lzs+7FUbmlpfrvHGIbBm08vZ3TFWpyhABx/XNN5pfkwfDjOVV9S2tuTmqIoL4c+\nfdr++F9RATt2pN83fDgUtXzwNQAvzAWbDc81P8ATL8vll8Ff/kLpxx/Auec2Hbt2LXi9TT9vg1KA\nF54Fj4eCK+aAx2PW5dxzsT/9NKV6Bcyc2XTOBjN4emZOM1/vzJNh6FByXn2JnIceNM9PtuaLxLf2\ntZrSjV/D1Kmtvw/plJdD377p0z6LFsGG9VBYCLW1lKz4FGYe3rS/ttYcs5/XNKGs6LOmTxkFG9aY\n9e0KFRXQuzfYOrFdEghAfb153dYYBuzaBf36pWzO5G8820idO64jM10S/0O11oZS6jLgMaAW2JS8\nP5aPvxI4vr2LVlf7OlAUU2lpPpWV9e0et2ZLNRu213KDz2yJ1k6aSjDpvIKxB+P6z6vsXbGG6ICB\nANhWraT42Bk0/PI3NP74hrTXtdTXUTJ5PNaamrT7wyNGUr1oWYubg+3rVZQsXUrg+BOps3sgVhbb\nGedT8pe/EPh//6Bu1gkA5N55B3l33tFq3RovvgRvowGN5jUcZ19I0dNP0/jwY3hHT0wcV/jJUpzA\nnoEjMGKvl3v+xeT9+Q/UPfYUgYvmpFw3Z8ESPID//ItwP/8sjQ8+hHfEuFbL0Zzj48UUnX4idff+\nncCF322x3/PgQ+QA9b+5jfz//TGBt96h7oLLzJ2hECXTJhEt60PN6++AxUJpaT6h+e9gt9uxhMME\nP1lKbQa/+31l3biBkpmH47vueny/+E2nXTf/Rz/E+fprVC9ZTrRP37THuF6YR8G1V1HzwquEZs0G\nMv8bzyZS58zPSSeTtMxOzJZ6XH+gPP6D1vpDrfVMrfWpmAF+M4BS6gTgV8BJWuvafSptF3n7060A\nTNq5CsNiIXTk9JT94YMnAKkf9V3z38QSjZLz+CMQSf/hw/XKS1hragjOmo3v6mtTvkKTJmPfsB7H\nogUtzounI/wXXZKyPaJGE5o8Bef772HduQPne/PJu/MOIoMGp1ybn/7U/P5HP8X3f79KuUZo2nSi\nhUU4P/ygaaNhYF/5JZHBQzEKmz5J+C+4GMNiIWduy9RM/L3wXXs9kQEDcb38L2hoaHFca5xvvm6+\nR2/8p+VOrxf3Ky8RGTQY/3cvJTJoMI7FCxIjlpzvvI1t6xYcyz5NpJOoqcH+5QrCk6cQGTTYLF9n\np4oA13vzsYRC5Dz5eKfNGbDs3YvrpReweutxPT+v1eOcCz4EIOexhzvldUXPlEnLfT5wK/CQUmoS\nsFNrnbi1KKXeBC7DTMOcBtyllCoE7gSO01pXdX6x91353gZWbNjL6DIXBV99RvjgQzCKilOOaQru\nKwieeDIAjoVmULbt2I7jow8IzT62xbXdc5/CsFio/+sDiRZ/nP3jJRSffgLuuU8RmjmraUcwiPuF\neUR79yb4nRNaXNN/8SXkL19K7t134nrtZQyXi7rHnyY8oakVnluaT0Nrd3mbjdC06bjeeh3rtq1E\nBw3GWr4T6969BKam3tSigwYTOuponB++j23DOiIjRjWV/6svMXJyiByk8F/4XfLu+hOu/7xK4IKL\n079uM/GbmmPJIvPmmJTicL32ChZfA/5rfwxWK6HpM3HPewbbqpVEDp6Q0g/gnvsU3gkT4aOPsESj\nBKfPxL76a1xv/gdrxS5zyYVOFP+9W/fuxTn/LYKnnr7f13S/9DyWUMj8/tmnaLzu+rSpqvgN1fnO\nW1h278YoK9vv1xY9T7std631YmC5Umox5kiZa5VSlyulzood8jDmDWAh5pDHPcAFmJ2tzyulPoh9\nDe6SGjQ2mjnMdsxfug2Ac117sASDhKbPbHFM+OBDgKSWeyCAY+nHRPPN/uN0nY42vQbH8qWEjj6m\nRWAHCB8xlfDwEbhe/zeW2qa0jfPtN7FWVeE/90JwtpwZGzjzbIycHHKefAxrdTXe2/+cEtgzEZph\n1jEeYOP1it/EksU7Y93PPpNUiAA2vZrw2HFgs+GPpVVynnwc26qV2FatxLp9W6uvb6mpTrS4rbU1\n2Fd9lbI/J3ZTjF83OOMoAJyLPsJasQvnu/MJjzuYSJ++uP71gjkO/v33zbrNnJVyM26XYWDZvTt9\nOffsSd0QieBYsrDV37ulrrbVT3EJPp95XNLru595CsNuJ3jUbOzr12Ff+mnL82LvOYAlHO6yEUrf\nNMvu3V07hyQSkXkPzWQ0QFhrfZPW+kit9Qyt9Qqt9RNa65dj+17SWk/UWh+qtX4mtu0fWuv+Wuuj\nk762dkUFCq68BEaPxlqxq9Vj6nxBFq/cRWmRmwmLzPRAPPAli/bpS7S0LBEsHJ8tw+L347/wYsKj\nDsL1xn+wVKd+EHHHxqfHg2MLFgv+iy/B4vebKQ2AYJDc++8xz2uW444z8gsInHamecx5F+K/5PJW\n69ea4PRYsIx9zI/XK11wD5x0KtHCIlzPzYX4SBS9Gks4THi8edOLDhlKcOYsHEs/oWT2kebX5PHY\nv/yixfUAHIsXYTEMwiPNTwLx1jCAbcM6HJ8sITTzaKKDzPt+KBbcHYsW4Hp+HpZIhMY5lxE4/yKs\ntTW43vwPvP8+hstFaPKUljfjNuT95pf0Hj+yxY3AsWgBvccOx5UURO2rvsJaU0Pg1NMJHToJ53vv\nYN1lZiJta1ZTcug4Cs86BWKt8HSKzjuD4iMPw1q+M1bGFdi/Xknw+JPwXXc9kL6xEH/P/aefheF0\nds0IpW+Yba2m16FjyL/2qi6rS+69d9PrkNFYt3VJmOmWuv3sj9DRx8DOneRf/b1EUGruw893EApH\nubpqKTkvPkfokEMJzjom7bHhgydg27EdS9VeHLGgGJoxyxw7HgzieumFpBcP4X7+WaLFxYmZoekE\nzr8Iw2pN/GfO+92vcXz+Gf5zzm9zolLDL36N9+ZbqL/zr61OMGpLZMxYor16mS13w0hquR/S8mC3\nm8A552Gr2IXz/XeB9C197+1/xnf1Nfi+fzX+s8/DYhi4n3wi7es7Fn0EgO9nN5k/L/yw6eVinxD8\nFzV1skb7DyA8bDiOxYtwz30Sw+UicPa5iRtgzt/vgxUrCE05AtzutH0k6bhefYnchx4wy7B4YWoZ\nF3xgXvvRh5q2xW5CoRlHpcwZsHjrKfjeHKz1dTg/Xkze79J3tNq/WoFj6SfYdldQcNUV5t9JrD/D\nf/EcQjNnERk4CNcrL6WOfkqqS2jmLAInnYpdr8H+2bI26/dt5376n1hCIdz/eh734490yWs4lizC\n0tiI87/vdsn1u6NuH9wbf/A/cNZZOBcvbHU1xE9X72bk3i1M+fvtRAuLqHvkn2lTIZCamnEsWmB2\nvE47Ev95F2LYbImZoADOd+dj3VOJ/5zzweVqtYzRvv0IHvsdHJ9/Ru6fbif3H38nfJAyg3YbogMG\nmiN0cjv4oBCrldCRM7Ht3IFt0wbsK78k2ru01VEaidTMM2YgStfSj4weQ8Ntf6ThD3dS/8A/iPTr\nj+vlF8HXcrSTc+ECDLebwCmnEx4+AsfHS8wbcDiM67m5RAuLCJx8Wso5oRmzsNbXYd+wnsDJp2IU\nlxAZOYrQ4VNxfPF57BizhR/t159or15tBnfb+nV4fnIdhsMRq1PqsfGfHcuXYVtjpkPiN6HQjKNS\n5gx4/vfH2Nevo/GyKwmPOojchx7A+dqrLV7TFfs0Fx51EI5PluD5zS9w/esFImV9CB7zHTPFdcHF\n5sSt/6Sen/yex298yX9z3U4wiPvFeURLSoj26oXn1zd1yc3KtmE90NSgEFkQ3LFY4PHHiQwdRu69\nd7dYMKuiykf1jt3c/MadWAN+6u9/iOiQoa1eLhQLZI6ln+BYvpTwhIkYRcUYffoQ/M4JOL5agfPt\nN7GtWU3OE2YrpPlol3Tix+Td9SeM3DzqHnu65ZjyLpDIY//nNWzbtpqBupVPAeGDDyE87mCc8980\n15/56ksMm43wmFaGPtps+C+8GGt9Ha7X/52yy7JnD/bVqwgdPg1cLjNoe+uxr/gc5/vvYqvYReDs\ncyEnJ+W85HRZ8vuanPaKp5uwWAiPn4Bt6+aU/oyEhgazpd3gpf5vD2Lk5rYa3CGWYguHcXy8hPCI\nkUT79ccoLCJwyunYN27A/fK/CE05Au8f/kzdo09h5OaSf/012Daub7qg34/7xeeIxIZvhkeMJOfR\nf2CtrSFw/kWJdfbj/QzuZiOUkt/z0Kxjmm6eK1ZgW7PafK00qQ1LTXXafgBLXW3Gk+EACAZT+wri\notEO5bSd77yNdc8e/OddRN3fH4VwmILvX4alam/bJzY0mP1pmWhsTPT9OBct7PZprM7S/YM7QGEh\ntY8+heF2k3/d1Vi3bE7s+nxtJde/fR+9Krfj+/ENBE84qc1Lhcebwd391BMtOl7jwabwkgsoOeoI\nnO+/R+jgQ4ikyWE3Fzz+RKK9egFQf899RA5S+1rLDom3cuNph7QpmTiLBf/Fc8yOvOfmYl+10iyn\n293qKf4LYkHq2dTWpWNxPLUxM+Vfx6IFiZZoun6KeOCODByUMroocPqZGLl5kJdH+NBJie2JT1or\nUztrAfLuuRP7mtU0XnkVgXMvIDxmHLa1axILlFkqK7HtKid49DFEe/XC/cKz2JctxeqtJxS/gSSV\nM9qrF3UPPwEOB5HRY6j/y9+weusp+N6liU8urrffMPP151+EUVRs3gRiN7Dk/pV4/4Xz48XYNqwz\nN0YiTe95Tk7TzdNbDxMnUnJQ8FbnAAAgAElEQVTUEZRMnYT7iUdT6mndvIleE8eQd9tvU98An4+S\naZMpOuPEjBdly7/+GkoOP6RFIM+95056jR+FdeOGjK4Tlxjue/ElhI4+Bt+Nv8C2fRt5d7T+zAFL\n1V5Kjp4Gs2a1ekwy28YNWGIB3Vq5G9tavU9lzFbZEdyByMET8P7xLqy1NRRceWnijzn/kQc5cv3H\nNE6dTsNNN7d7nejQYUQ9+dhiHWHJLcng8SfScNPNNF7xffPrez/A+5cMl81xOqn7+6PmZJ6zzm3/\n+E4SGTmKSFmfRH3SdaYm859zPobTSe4Df8Xia0jc7FoTHT6C4JEzcC78COvmTYntzoXmx+Ng7OYY\nPNL81/XvV3C+/QbhsePTjv4xysqou/8h6h58JGXYpOHJp+7hx2Hu3JSUWqsjZmJ57mhJCd7f/j5x\nrCUSwb7m65RzQpMOw3/uBVj37MFzq/k3kvx7D804Cu/v/kDtvJeI9h+Q2B449wIaL7sS+9cr8fzi\nZ0BTSzweyCNjx1H79PPU33M/kVEHpRQxfox73lwAbJs2tnjPG394Hb5rfgzXXEPj5VdiOBzkPPFo\nSuvU/exTWHw+3E//M6W16/rPq1grd+P4bDme3/yixXvdnGX3blyvvoS1qiqxfHT8vcx57GGzz+m9\n+e1eJ866qxznu/MJHTop0bfk++mNRIuLcf73nfQnRaPkX3sVti2bYenSREd2W+I3x/DoMUBq305P\nljXBHczWQeNFc3B8+QWeX/+CwAcfcdprD1GXX0LDw09k9ug5qzWxzoxhsxGaemTTPpsN3w0/x/un\nu82vP95F+NDJGZcvdPQxaWdpdimLJdF6h/aDu1HSi8CJp2Dduzej4yE5SDUNo3Qs/IhonofwRLOV\nbZSVER49BseXX5ijQS6e02p6KHD+RYSnTmuxPfidE+H01PHmrY2YSfSHnHtB4pNH82PtK5s6mOOf\nyhzLzQVP4zcjACwWGn94HeFDDm1RJu/v/0ho4qHkPPs0uXfegeOD/xKackRKIA/NnIX/u5e2rOcp\npxMtKDRHKEUiafs4jOISGm75PTzwAN4/30Pw+JOwr16FfYXZ/0Akkrg5WOtqcb3xWuLc+Kep8LDh\n5DzxaMqIoHTcLz6XWLMn+ZOY87/vYq00h5Emj3hqj+v5eVii0dS0pc1m9gNt25ryCTsu969/wfXe\nO0Q95qzLdJP/mrOvN4N742VXmuXdhzJms6wK7gDeO/5CeOx4cv75KH0uvxCAT3519z4tVRv/zxWe\nOAnD0/3Xtkh0QHryiQwd3u7x/oub0gdtpnFiAqeeQdSTj/u5udjWrcW+9BPs69cRmjoNYh2ZQCLF\nZTgc+M+5YF+rkVZk+AiM3LxEoI5LpAMuTK5Lais/eTRQZOw4QhPN4B0ePSbziUMuF3WPPEm0qIi8\nO+/AYhitD4ttLieHwFnnYttVjvOD99oezRQT/93EPyE4PnwfW/lOgjOPjm03g7J100acixYQnD6T\numdfJOrJJ/9n17e+MFtsATfD6SQ49UgcKz7HFkt1xV8rmufBsWRh+2P8k6/ndhM465yUXcHYpyJn\ns8Dt+PB9cv90O5EBA6l79ElzW/PgHgy2mCFtW7fW3HXsd4gMHJQyy3l/WerrMqvvvohGsXi7flmF\nrAvu5OZS99iTRPMLcPi8PDljDv3PbjvP3lz8P1dyi7c7i3eqhscfnNHa5/GOvMQ57cnLI3DWOdh2\nbKdk+mEUn/Id8zrTU9+/4Awzhxo88RSMWP/Dfot90rKt1dhiLThLRQXOd94mdMihRJLKHx49FsNm\na2q5f7WCaFFRYpx9vIWZboJbW6KDh1B/v9mnYeTmEjjjrHbOaNIUrJ9uarm38Z4HZx9nTup66UVo\nbEy0sBt++WuCU4/EueADrFu34H4uPtR0DpHhI6n/2wNYfD4KvjenxfBLwHxwi15D4KRTafzhdWaZ\n5j2NZfdunO+8RWj8BAKnn4m1puVktHTsn35ijng65fSUpS6g6e8iPtTYrFiQ/OuuBruduoefIHTU\n0VBUlJijEZd/7VWUTJuU0kls27AOw+UyZ1pPn4m1uhrbqpXtlrE9Fm89JZPHU3jxuZ0a4HMefYhe\nY0dktALq/si+4A5Eho9k9zMv8o9jr2LpiXMoK8pp/6QkgTPPoeHnv8QX+yPv7qJDh1F/933mx/tM\n2GzU//0R6u77fy3+Y7bGd8PP8X3/ahovuYLGS67Ad/W1KZ8AINZn8Ytf4/3tbftahbZf+6r/wRKJ\nmIGroQH3C+YEqBYTxNxuIgeNxr56FZbaGuwbN5j57Vh6yH/RHBpuuhnfj366z2UIHn8SdX9/hLoH\nHt6nT3vhiZMIjxmL863XsX/+WYt1f1qw2wlccDHWulrcz/wT15v/IaxGE550WNNQ1rlP4p43l6gn\nn8CpZ5jlO+1MfFdfg33dWvJ/9uMWI0oSndwXzSH4nRMSy07nzH0ykUZLTDLLIO3RtG5Sy0l6ETWa\naO/SxPwLiK0jVLGLxsuvJHzY4WZ/y6xZ2LZsToyEse7cgeu1V7DtKscRH05pGNjWrycybDjYbCmz\nnPeXY8kirDU1ON9/j9w2Fu3b5+suXoTF78f91OOdds10sjK4A3xWNJzXDjmZQ0d3YF0Otxvfz27q\nvNblt4B/zmWEJx2W8fGhI2dkvH4MmGPyG/5wJ967/ob3rr/RcNsdGCXN3j+Hw+xQGzwk4+tmInj6\nWTReeRX2NavJ//lPzXRAbAJUc+GDJ2Dx+XDFxqendBi73fhu+HlKp+m+CJxzPsFTTmv/wGQWC/6L\n5mAJhbDW1WbYx2H223hu/TWWYND8xGGxmOmxPA+5D9yLbecOs+M+aY5Ew69/R+iww3G/9GLqiBuf\nD1fsgTGhWbPB4cB/3oVYq6rIvetPGE4ngbPPS5pB3E7gjC8IN3hI+k+/FgvBGTOx7SpPDCNN3Ay+\ne1nTcbPNFTEdsc559/PPYomlW+LbrLsrsHrriYw0+ziSZznvr/hNzMjNI+/uP+Pch87ktsQ7gPfl\nmQ0dkbXB/Yt15nohh44qPcAlEd8E7y23E5o0GfcL87CvW0vglNNaLAwHTXn3eB45k2Da1fznXogR\n6+zPpDyREaMITj0SSyCAYbebncYAHg+BM8/GEgsYzT854XRS98g/E5OJHO+/h3XTRnKe+SdWbz3+\nCy9OjFCKt7gtgQCBk07FKOlFtF9/wiNG4liyuNXZ4JC0INyF3201DdiUmvmoaVTNxEOJjE2aUxEL\n7s6FH5k5/LlmDt+wWBLBPZ6Kiy9xER0wMDHLua0yZsKxaAGG00nNcy9jOJ3kX/ODNtdSykgkgi32\n8BtrVRXO+V33IPusDe5bK+pxO20M7tP1E4XEt4DLRd3D/yRabAb01iaWxftTHMs+Tfn5QDJ69yZ4\ngrkKaXhCZuWJp2CC3zkxpfM3Xu94qqa5aP8B5mSiUIiiC86i1xET8fzq/8xzkzqfI6PHEJp8WOya\nTdtD049KTEZrTWJBuDY++SXPe0g7qgZg/HiiJSU4Fi3AsWQRts2bCJx2JuGDDzFHNfl8ic7UyIiR\nSdc+ypzl/NlyOspSXYX9qxWEDjuc8BFT8d7+Z6zV1eT/7PoOXxPAum0rlkCAUGzkVfNJbJ2pIw/r\n+NYLR6LsqvIxtF8+lg6sySK6p+igwdTOfRHHwo9Sl1dOktxZabjdREaOSnvcN817y+8Jjx1H8OiW\nS0qnEzj7PBq2bTWXvkgSnnK4OTzz0MmtDjUNHX0MdY8+hfPdt5vOmzCR6NBhqWX60904PvrQXL8p\nfu6MmeQ8+RiORQsIT57S4trxBeGCs2YnOqrTiQwfSaRff5yLPsK+8ktzVE3zNJrVSmj6Ubhee4W8\nO8x+Gv/Fl+B8520cX35hrt8TS3Ek/x4Dp5xGzlNP4H5hHt7Dj2i1DG1xLFmMxTASnev+S6/A/fyz\n5qedrVs6nFq0x8obPOkUsNsSz2zoaCqwLVnZci/f6yMSNRhYKq32niY8eQqN1/9vq+kAo6CQSCyI\nhceOy2zuwzcgOmQovht/kXl5nE58N/6C6PARqdstFhqvuobwlLaDWvDU0/H+9YHEl/97P2hxTHjC\nRHPN+aT3Mj7+v/kolrjEgnDtDQe1WMyRLXv2tDqqBpomwTk+WUJkyFBC06antPrjaZnk4J5YtuGl\nF9KueZSJeL9Cos/AYqFxzmXmQnnPze3QNaFp2GZ45Cj8F1+KJRrFneZB9p0hK4P79kpzqJcEd5FO\nPBUTX8pYZC4xGe3Tj7Fu2oh129amry2bmxaEO+nUdq+V3Nna2s0g5ZiL5pit+alHYthsOBd+hH39\nOqK9S1NvDMlrHiVN6jIv0soyDM22Oxd+hJGTQygptRU4zVwCwz3vmdRx9JFI+uWfo9EW6+PY1psd\nyJERoxLPbEj3IPvOkOXBPa+dI0VPlJik9i3oTO2OgjOOwtLYSK8jJtJr8vimrykTzAXhzjmvzfWI\nkq8DmKNqWplbEBl1EJGyPik5fMOTT3jiJOyfL8e6bSvhZss6QPo1j9xzn6L3qEEt1uZxLPiQ3qOH\nkneLufSEpbIS++qvCU2Zmrraq8eD/8yzsW3bmujQxeul6ITZFM+amvqsh0iEgksvpNfkcSmTrmwb\n1mFYLObku9gzG2ybN3XKuPzmvh2fSTvZjkrzzRwgLXeRRuOlV2BpaGiRrxaZabzqGix+P5Y0z5Y1\nXG58rTxIvrno4CHU3/EXIqPHtD65zmLBe/e9WKqqUp50FppxVGKpiHT9JtHhIwhOm45zwYdYt2zG\nWluD5/9uwBII4PnVzwkfPIHw5ClYy3dScPX3sPh85D54L+GJhyb6KtI90Md/0SXkzH0q8djM/Bt/\ngiP2sJr8666m7qnnwGol9547cc1/CzBTSqFjjgPM0T3RQYMTq6E23HQz0b79zHH6nSwrg/v2Si9F\nHieeHEf7B4sexyjpRcOvftv+gSKt6NBheO++r1Ou5b/yqnaPCR7fcoZ5cPpMcv92F0DKM39Trn3R\nHJxLFpHz0AO45r+NJRCg4Sc/I/dvd1Hw/cuofut9Cq66AuueSnw/vA73U0/g+emPCE853HyNNGP0\nw4cfQXjESFyv/5vwvXfj/tfzhCZPwfB4cL3zNjn33UP4kEPJvfMOjJwc8wEiixYQOuY4LPV12Cp2\nEUx6DnN04CAabr4lk7dqn2VdWsbnD1FVF5B8uxBZLHT41MQDWCIjR6Y9JnDameakrkcewrZ1Mw03\n3Ijvl7/B93+/MpfKOHoqjk+W4D/jbBpuvR3vPfdhbfDi/OC/5qJ3aRaKMyedXWJ+Arj9VqIlJdQ9\n8k/q/v4okf4DyLvjNgp+cDnY7dTOfRHDbk90zjYfk9/Vsi64b4+lZCS4C5HFcnMJxYZihke2zLkD\niTWPAIIzZ+G78ZcA+H7yMwLHfgfr3r2ER47Ce8995gzfM8/B9/2rAVosepcscMFFGDYbhsVC3YOP\nEB0wEKN3b3Otf6sVa20N3tv+SGj6TMKTDsP+xedY6mqbRva08kmjs2VdWibemTpAOlOFyGoNv72N\n4MdLiLaRr/bd8HOMPA++6/+36fkAViv1Dz5M+O/347/wuylrATXccjtGQUFiUlk60T598f7lbxgO\nRyKXDhCecgR1TzyDbctm/Fd8HzBXwHR8+jGOpIeyNF/Xv6tkYXCXlrsQPUF48pS0E6mSRQcOouG2\nlot+GcUl+H6Z5gHnTie+m37d7munW58fWvYPhKYfBXffiWPhAmw7tgPpO4C7QhYGdy9Wi4X+vTv4\nUGkhhOgkocMOx3C5cCz8CEskQjTPQ7Rvv2/ktbMquBuGwY7KBvqU5OCw29o/QQghulJODqHDDsex\neCE4HIRHj211WYjOllUdqlV1ARoDYRnfLoT41ghNn4nFMLAEg62O7OkKGbXclVL3AFMBA7hea700\nad8ZwM1AAJintb6/vXO6isxMFUJ82wRnzCLvz38ASKw7/01ot+WulJoFjNJaTwOuBO5N2mcF7gdO\nBo4CTlNKDWzrnK4ka8oIIb5twpMmY8QemvJNrkKaSVrmWOAVAK31aqBYKVUQ29cbqNFaV2qto8B7\nwHHtnNNldiRGykjLXQjxLeF0Eoqt0hn+hsa4Q2Zpmb5A8qr3lbFtdbHv85VSo4DNwGzgg3bOSau4\nOBf7fnSClpbmU+sLYbXAmJFlWK09Yx330tLMn9eZLXpinaFn1jtr6vznP8Lrr1Ny9LR2H1LfWXXu\nyGiZRNTUWhtKqcuAx4BaYFPy/nTntKa6umPrLoP5ZlRW1tPoD2G3Wdm7t+XT3bNRvN49SU+sM/TM\nemdVnYeNgevGwN6GNg/rSJ1buxlkEtx3Yra64/oD5fEftNYfAjMBlFJ3YLbg3W2d01XCEQObLasG\nAAkhRIdkEgnnA+cCKKUmATu11olbi1LqTaVUmVIqDzgNeLe9c7pKJBrFbusZ6RghhGhLuy13rfVi\npdRypdRiIApcq5S6HKjVWr8MPIwZzA3gDq31HmBP83O6rAZJQuEodmm5CyFEZjl3rfVNzTatSNr3\nEvBSBud0uUjUkJa7EEKQZTNUwxFpuQshBEhwF0KIrJRVkTAckbSMEEJA1gX3qAyFFEIIsii4R6JR\nDAMcEtyFECJ7gns4YgBgk7SMEEJkT3CPRKIA2NtZt0EIIXqCrImE8Za73Z41VRJCiA7LmkgYjrfc\nJS0jhBBZGNwlLSOEENkU3GNpGWm5CyFENgX3eFoma6okhBAdljWRsKnlnjVVEkKIDsuaSBhvucs4\ndyGEyMLgLjNUhRAiq4K7zFAVQoi4rAnuEelQFUKIhKyJhCEJ7kIIkZA1kTAi49yFECIha4K7jHMX\nQogmWRMJJbgLIUSTrImEsvyAEEI0yZ7gHo1PYsqaKgkhRIdlTSSMt9xlEpMQQoA9k4OUUvcAUwED\nuF5rvTRp37XAHCACLNNa/0Qp1R94DHABNuCnWuvlnV34ZOGwrOcuhBBx7TZzlVKzgFFa62nAlcC9\nSfsKgBuBmVrrGcBYpdRU4AbgZa31bOAm4PauKHwyScsIIUSTTCLhscArAFrr1UBxLKgDBGNfHqWU\nHcgFqoA9QK/YMcWxn7tURNIyQgiRkEkk7AtUJv1cGduG1toP3ApsBLYAn2it1wL3ABcopdYADwO/\n6cxCpxOSVSGFECIho5x7M4noGWvB/xI4CKgD/quUOgQ4DXhea327UupU4C/A2W1dtLg4F7vd1oHi\nmBwOsyplpfmUlno6fJ3uprQ0/0AX4RvXE+sMPbPeUueOyyS47yTWUo/pD5THvh8DbNRa7wFQSi0A\nJgPTgZtjx7wDPNjei1RX+zIsckulpfl4GwIA1NX4cGJ0+FrdSWlpPpWV9Qe6GN+onlhn6Jn1ljpn\nfk46maRl5gPnAiilJgE7tdbxV98MjFFK5cR+PgxYB6wHjohtmxLb1qUSM1TtknMXQoh2W+5a68VK\nqeVKqcVAFLhWKXU5UKu1flkpdSfwvlIqDCzWWi9QSq0HHlVKnR+7zI+7qgJx8pg9IYRoklHOXWt9\nU7NNK5L2PQQ81Oz4cuDk/S7dPkg8Zs8qHapCCJE1zdzEY/YkLSOEENkU3GOP2ZOWuxBCZE9wj0Si\n2KwWLBYJ7kIIkTXBPRSJykgZIYSIyZpoGIkY2CUlI4QQQBYF93AkKsMghRAiJmuiYThiSHAXQoiY\nrImGZstd0jJCCAFZF9yzpjpCCLFfsiYahqOSlhFCiLisiYbhsKRlhBAiLiuCu2EYRKKGPGJPCCFi\nsiIahhOP2JOWuxBCQJYE91A4AsjDsYUQIi4roqGs5S6EEKmyIhomnsIkaRkhhACyJLiHwvHgnhXV\nEUKI/ZYV0VBa7kIIkSo7gnus5S4dqkIIYcqKaBhPyzgkuAshBJAlwT3xcGxJywghBJAlwT0Uz7lb\ns6I6Qgix37IiGsZz7vKYPSGEMGVFNAzJaBkhhEiRFcE90XKXtIwQQgBgz+QgpdQ9wFTAAK7XWi9N\n2nctMAeIAMu01j+Jbf9ZbHsIuCb5nM6WGOcuaRkhhAAyaLkrpWYBo7TW04ArgXuT9hUANwIztdYz\ngLFKqalKqXHAhcBhwNXAqV1R+LjEDFWrpGWEEAIya7kfC7wCoLVerZQqVkoVaK3rgGDsy6OU8gK5\nQBVwFvC81joMfBb76jJNM1Sl5S6EEJBZcO8LLE/6uTK2rU5r7VdK3QpsBBqBeVrrtUqpoUBEKfUW\n4ABu0FqvaOtFiotzsdttHakDYV0JQElJLqWl+R26RnfV0+oLPbPO0DPrLXXuuIxy7s0kch+xtMwv\ngYOAOuC/SqlDYsfYgJOA6cAjwJS2Llpd7etAUUzx0TI+b4DKyvoOX6e7KS3N71H1hZ5ZZ+iZ9ZY6\nZ35OOpkE952YLfW4/kB57PsxwEat9R4ApdQCYDJQAazRWhvAwlhLvsuEZG0ZIYRIkUk0nA+cC6CU\nmgTs1FrHby2bgTFKqZzYz4cB64A3gRNi54wGtnVimVuQx+wJIUSqdlvuWuvFSqnlSqnFQBS4Vil1\nOVCrtX5ZKXUn8L5SKgws1lovAFBKnaSUWhK7zLVdVH5AHrMnhBDNZZRz11rf1GzTiqR9DwEPpTnn\nt8Bv96t0GZLH7AkhRKqsiIbysA4hhEiVFcFdHrMnhBCpsiIaJtaWkZa7EEIA2RLcZYaqEEKkyIpo\nGJLgLoQQKbIiGkpaRgghUmVFcA9FZIaqEEIky4poGG+5OyS4CyEEkCXBPRSOYrGAVdZzF0IIIEuC\nezgSlc5UIYRIkhURUYK7EEKkyoqIGApHZaSMEEIkyYrgLi13IYRIlRURMSwtdyGESJEVwT0kLXch\nhEiRFRExHI5is2ZFVYQQolNkRUQMRQwcdknLCCFEXFYE93A4IksPCCFEkm4fEaNRg6gBdpmdKoQQ\nCd0+uMta7kII0VK3j4jycGwhhGip20dEeTi2EEK0lEXBvdtXRQghOk23j4jhqKRlhBCiuW4fEeUR\ne0II0ZI9k4OUUvcAUwEDuF5rvTRp37XAHCACLNNa/yRpXx9gDXCW1vqDTix3QlgesSeEEC20GxGV\nUrOAUVrracCVwL1J+wqAG4GZWusZwFil1NSk0+8ENnZukVNFYmkZecSeEEI0ySQiHgu8AqC1Xg0U\nx4I6QDD25VFK2YFcoApAKXUMUA981dmFThYKx1vukpYRQoi4TNIyfYHlST9XxrbVaa39SqlbMVvn\njcA8rfVapZQT+C1wBvDXTApSXJyL3W7bp8ID7KzxA1CY76a0NH+fz+/upM49R0+st9S54zLKuTeT\naCLHWvC/BA4C6oD/KqUOwQzqD2uta5RSGV20utrXgaLA3qoGAAKBEJWV9R26RndVWpovde4hemK9\npc6Zn5NOJmmZnZgt9bj+QHns+zHARq31Hq11EFgATAZOAK5TSn0MnAI8qJQat08lzlDTaBnJuQsh\nRFwmEXE+cC6AUmoSsFNrHb+1bAbGKKVyYj8fBqzTWk/XWk/VWk8FXgeu0Vqv6tyim2ScuxBCtNRu\nWkZrvVgptVwptRiIAtcqpS4HarXWLyul7gTeV0qFgcVa6wVdW+RUsvyAEEK0lFHOXWt9U7NNK5L2\nPQQ81Ma5l3eoZBmStIwQQrTU7SNiPC0jQyGFEKJJ9w/usbSMTGISQogm3T4iRiLxlnu3r4oQQnSa\nbh8RQ9KhKoQQLXT74B6JB3drt6+KEEJ0mm4fEROP2bN3+6oIIUSn6fYRUca5CyFES9kT3CUtI4QQ\nCd0+IkpaRgghWur2EbGp5S5pGSGEiMua4C7j3IUQokm3j4jxSUwOScsIIURCt4+I8UlMNknLCCFE\nQrcP7olJTJKWEUKIhG4fEROjZWScuxBCJGRBcI9it1mwWCS4CyFEXBYEd0NSMkII0Uy3j4rhaFSC\nuxBCNJPRY/a+zWZO6A+y9IAQQqTo9sH9+CmDKC3Np7Ky/kAXRQghvjWkySuEEFlIgrsQQmQhCe5C\nCJGFJLgLIUQWkuAuhBBZKKPRMkqpe4CpgAFcr7VemrTvWmAOEAGWaa1/opSyA48CI2Kv8TOt9cLO\nLrwQQoj02m25K6VmAaO01tOAK4F7k/YVADcCM7XWM4CxSqmpwCVAQ2zblcDdXVF4IYQQ6WWSljkW\neAVAa70aKI4FdYBg7MsTa63nAlXA08ANsWMqgV6dWWghhBBtyyQt0xdYnvRzZWxbndbar5S6FdgI\nNALztNZrY8eFYv/+BJjb3ouUlubv18pfpaX5+3N6t9UT690T6ww9s95S547rSIdqIgjHWvC/BA4C\nhgFHKKUOSdp/LTAJ+N1+llMIIcQ+yCS478Rsqcf1B8pj348BNmqt92itg8ACYDKAUupK4DTgTK11\nCCGEEN+YTIL7fOBcAKXUJGCn1jq+kMtmYIxSKif282HAOqXUcOCHwNlaa3/nFlkIIUR7LIZhtHuQ\nUuqPwFFAFLgWOBSo1Vq/rJS6GrgCCAOLtdY/V0r9AbgQ2Jp0meNjrXshhBBdLKPgLoQQonuRGapC\nCJGFJLgLIUQW6vYP62hraYRsopT6MzAT83d2B7AUeAqwYY5eukRrHThwJew6sQ77lcBtwHtkeb2V\nUt8Ffo7Zj/Ub4Euyv84e4EmgGHABtwK7gL9j/t/+Umv9PweuhJ1LKTUeeBW4R2t9v1JqEGl+x7G/\nhZ9g9nf+Q2v9aKav0a1b7m0tjZBNlFKzgfGxep4I/BVz7sADWuuZwHrgewewiF3tZsyZz5Dl9VZK\n9QJ+C8wATgXOIMvrHHM5oLXWszFH5/0N8+/8eq31dKBQKXXSASxfp1FK5QH3YTZU4lr8jmPH/QY4\nDjga+KlSqiTT1+nWwZ22l0bIJh8B58W+rwHyMH/Z/45tew3zDyDrKKVGA2OB12Objia7630c8K7W\nul5rXa61vorsrzPAHpqWKSnGvJkPS/oknk31DgAnY84hijualr/jI4ClWutarXUjsAiYnumLdPfg\n3hdzOYS4+NIIWUVrHZhsq98AAAIRSURBVNFaN8R+vBJ4A8hL+mi+G+h3QArX9e6iaZ0iyP56DwVy\nlVL/VkotUEodS/bXGa31PGCwUmo9ZmPmZ0B10iFZU2+tdTgWrJOl+x03j2/79B509+De3H6tT/Nt\np5Q6AzO4X9dsV1bWWyl1KbBEa72plUOysd4WzBbs2ZipisdJrWc21hml1Bxgq9Z6JHAM5uKDybKy\n3q1ora779B509+De1tIIWUUpdQLwK+AkrXUt4E2aGTyA1I942eIU4Ayl1MfA94Ffk/31rsCcDBjW\nWm8A6oH6LK8zmOmGtwG01iuAHKB30v5srXdcur/r5vFtn96D7h7c21oaIWsopQqBO4FTtdbxjsV3\ngXNi358DvHUgytaVtNYXaK2naK2nAo9gjpbJ9nrPB45RSlljnasesr/OYHYiHgGglBqCeVNbrZSa\nEdt/NtlZ77h0v+NPgClKqaLYaKLpmOt3ZaTbz1BtvjRC7K6fVZRSVwG3AGuTNl+GGfDcwBbgimxe\noE0pdQvmWkZvYw6Zy9p6x5b0uDL24+8xh71me509wGNAH8zhvr/GHAr5EGYj9BOt9Q2tX6H7UEpN\nxuxLGoq5NPoO4LvAEzT7HSulzsV8IJIB3Ke1fibT1+n2wV0IIURL3T0tI4QQIg0J7kIIkYUkuAsh\nRBaS4C6EEFlIgrv4/+3UgQwAAADAIH/re3wFETAkd4AhuQMMBQx58Fx045OkAAAAAElFTkSuQmCC\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "metadata": {
        "id": "-AGE67k2lGxL",
        "colab_type": "code",
        "outputId": "e1ce36c6-50c0-4b3a-81ea-4fa7861d5e8b",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 282
        }
      },
      "cell_type": "code",
      "source": [
        "plt.plot(H.history['loss'])\n",
        "plt.plot(H.history['val_loss'],'r')"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[<matplotlib.lines.Line2D at 0x7fd975d12cc0>]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 113
        },
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXIAAAD4CAYAAADxeG0DAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJzt3Xl4HNWd7vFvLd0ttXbbMt7NZg47\nmC02+5Z9H8gy2SCBkIWZkBmSGQhZJ8lNMhmGhAyZhMyEDIQkd5JcCARCCATC4gDGLDZgH1YbsGRb\n3rSrl6q6f1RLasmyLQvJdqnfz/PwIHVXd58jyW+d+tWpOk4URYiISHK5e7oBIiLy2ijIRUQSTkEu\nIpJwCnIRkYRTkIuIJJy/uz+wra1zzNNkmpqybNnSM57NSYRK7Hcl9hkqs9+V2GfY9X43N9c523su\nUSNy3/f2dBP2iErsdyX2GSqz35XYZxjfficqyEVEZFsKchGRhFOQi4gknIJcRCThFOQiIgmnIBcR\nSTgFuYhIwiUmyDd39PGz3z9NLh/s6aaIiOxVRnVlpzHmKmAREAGXWGuXlh6fDdxYtun+wGXW2l+M\nd0MfXbWB397zPHOnZTnygGnj/fYiIom10yA3xpwGLLDWLjbGHAL8FFgMYK1dC5xe2s4H7gVumYiG\nOm58dWqhGE7E24uIJNZoSitnATcDWGtXAk3GmPoRtjsf+K21tmv8mjfI9+KmFgOtaCQiUm40pZUZ\nwLKy79tKj3UM2+5C4A07e7OmpuyY7jHQ1FANQLYmTXNz3S6/PunU58pRif2uxD7D+PV7LHc/3OYO\nXMaYxcAqa+3wcN/GWO9y1tuTj1+/tZe2ts4xvUdSNTfXqc8VohL7XYl9hl3v945CfzSllRbiEXi/\nWUDrsG3eBtw16haNgefF+49ioBq5iEi50QT5ncC5AMaYY4AWa+3w3cjxwJPj3LYhVCMXERnZToPc\nWrsEWGaMWQJcDVxsjDnfGPPuss1mAhsmqI0A+KUReaARuYjIEKOqkVtrLxv20JPDnj9i3Fq0Hd7A\niFxBLiJSLjFXdvpuf41cpRURkXLJCfL+EXmoEbmISLnEBXmgEbmIyBCJCXJNPxQRGVliglzTD0VE\nRpacIHc1/VBEZCSJCfKB6YehRuQiIuUSE+S+auQiIiNKUJBr1oqIyEgSFOSlhSU0IhcRGSIxQe65\n/SNyBbmISLnEBLnrOriuo+mHIiLDJCbIIa6T62SniMhQiQrylKcRuYjIcIkKct93CXTTLBGRIZIV\n5CqtiIhsI4FBrtKKiEi5xAW5ph+KiAyVqCBP+RqRi4gMl6gg9z1HKwSJiAyTsCB3da8VEZFhkhXk\nvksQRoSRwlxEpF+yglx3QBQR2UYig1xzyUVEBiUqyFN+aUSuVYJERAYkKsg1IhcR2VbCglzLvYmI\nDJewINfJThGR4ZIV5L5KKyIiwyUqyFMDNXKNyEVE+vmj2cgYcxWwCIiAS6y1S8uemwv8EkgDj1lr\nPzkRDYXBWSu6TF9EZNBOR+TGmNOABdbaxcAFwNXDNrkSuNJaewIQGGPmjX8zY6qRi4hsazSllbOA\nmwGstSuBJmNMPYAxxgVOAW4pPX+xtfblCWqrauQiIiMYTWllBrCs7Pu20mMdQDPQCVxljDkGuN9a\ne/mO3qypKYvve2NrbGlEXlNbRXNz3ZjeI6kqrb9QmX2Gyux3JfYZxq/fo6qRD+MM+3o28H1gNXCb\nMeat1trbtvfiLVt6xvCRsf4g37y5m7a2zjG/T9I0N9dVVH+hMvsMldnvSuwz7Hq/dxT6oymttBCP\nwPvNAlpLX28E1lhrX7DWBsDdwGGjbtkuSvVfEKRL9EVEBowmyO8EzgUolU9arLWdANbaIvCiMWZB\nadtjATsRDQXVyEVERrLT0oq1dokxZpkxZgkQAhcbY84H2q21NwGfBX5WOvG5Arh1whqre62IiGxj\nVDVya+1lwx56suy554GTx7NR26PphyIi20rUlZ0qrYiIbCtRQa5L9EVEtpWoIB8oregSfRGRAckK\ncr//fuQakYuI9EtWkGvWiojINhIZ5Jq1IiIyKFFBntKsFRGRbSQqyAdKK7pEX0RkQDKDXCNyEZEB\niQzyQEEuIjIgWUGu6YciIttIVJCnVFoREdlGooJ88MpOjchFRPolK8g1/VBEZBuJCnLPVY1cRGS4\nRAW54zj4nqNZKyIiZRIV5ACe52pELiJSJnFB7rsORd3GVkRkQPKCXCNyEZEhEhjkqpGLiJRLXJDH\nNXIFuYhIv8QFuUorIiJDJS/IXUdrdoqIlElckGv6oYjIUIkLct9zKBZDokhhLiICiQxylwgIFeQi\nIkACg9zzdL8VEZFyiQty39UqQSIi5ZIX5BqRi4gMkcAg1z3JRUTK+aPZyBhzFbAIiIBLrLVLy55b\nDbwCBKWHPmitXTu+zRw0UCPXKkEiIsAogtwYcxqwwFq72BhzCPBTYPGwzd5sre2aiAYON7Dcm0bk\nIiLA6EorZwE3A1hrVwJNxpj6CW3VDvSf7FSNXEQkNprSygxgWdn3baXHOsoe+5ExZl/gAeBya+12\nU7apKYvve2NoaqyuLhP/v76K5ua6Mb9P0lRSX/tVYp+hMvtdiX2G8ev3qGrkwzjDvv8ycAewmXjk\nfg7wm+29eMuWnjF8ZKy5uY58rgjAxo3dNFaNpfnJ09xcR1tb555uxm5ViX2Gyux3JfYZdr3fOwr9\n0SRhC/EIvN8soLX/G2vt9f1fG2NuB45gB0H+Wg1OP1SNXEQERlcjvxM4F8AYcwzQYq3tLH3fYIz5\nozEmXdr2NOCpCWlpidc//VB3QBQRAUYxIrfWLjHGLDPGLAFC4GJjzPlAu7X2ptIo/CFjTC/wOBM4\nGgddECQiMtyoiszW2suGPfRk2XPfB74/no3aEV2iLyIyVAKv7NSIXESkXOKC3NMl+iIiQyQuyPtH\n5IEu0RcRARIZ5BqRi4iUS1yQe7pEX0RkiMQF+UBpRSNyEREgkUGu0oqISLkEBrmmH4qIlEtckOsS\nfRGRoRIX5IM1co3IRUQgiUHuqkYuIlIucUHuqUYuIjJE4oJca3aKiAyV2CAv6hJ9EREggUHuaYUg\nEZEhEhfkqYHSikbkIiKQwCDXmp0iIkMlLsg9TT8UERkicUHuug6u4+hkp4hISeKCHOLyiqYfiojE\nEhnknufqgiARkZJEBrnvOaqRi4iUJDTIXU0/FBEpSWSQe66j29iKiJQkMsh91chFRAYkNMg1a0VE\npF8ig1yzVkREBiUyyDVrRURkUDKD3HUJwogo0qhcRCSZQd6/bqcu0xcRGV2QG2OuMsb81RizxBhz\n/Ha2+ZYx5t5xbV0Z/5GH4aijcF99Bc/TjbNERPrtNMiNMacBC6y1i4ELgKtH2OZQ4NTxb94g/5mn\nYPlyUn99cHCVIJ3wFBEZ1Yj8LOBmAGvtSqDJGFM/bJsrgSvGuW1DhLNmAeC2tgyWVjQiFxEZVZDP\nANrKvm8rPQaAMeZ84C/A6vFs2HDBzNkAeC1ry+5JrhG5iIg/htc4/V8YY6YAHwXOBmaP5sVNTVl8\n39v1Tz3qYACqN22gtiYNQH1jNc3Tanf9vRKoubluTzdht6vEPkNl9rsS+wzj1+/RBHkLZSNwYBbQ\nWvr6TKAZuB/IAAcYY66y1v7D9t5sy5aesbU0StFcVUXhpTUUCwEAG9q6SFXAFMTm5jra2jr3dDN2\nq0rsM1Rmvyuxz7Dr/d5R6I+mtHIncC6AMeYYoMVa2wlgrf2NtfZQa+0i4N3AYzsK8dfEcWDOnLi0\nohq5iMiAnQa5tXYJsMwYs4R4xsrFxpjzjTHvnvDWDTdnDu7GNtJhEVCNXEQERlkjt9ZeNuyhJ0fY\nZjVw+mtv0g7MmQNAQ/tGQPPIRUQgaVd2zp0LQP2WeBKNSisiIkkL8tKIvG7zOgAKKq2IiCQ1yDUi\nFxHpl8ggr928HoCibpolIpKwIC/VyLObSkFe1IhcRCRZQT5tGlE6TW0pyNu783u4QSIie16ygtxx\nCGfOomZjHOQtm7r3cINERPa8ZAU5EMyajb+pjUxUpHWjglxEJHFBHs6chRNFLPD7aNnUreXeRKTi\nJS/IZ8czVxbQSW8uYGuX6uQiUtkSF+RBaYGJ+cV2AFpUXhGRCpe4IA9LC0zM6tsCKMhFRJIX5KUR\n+dTOTYBmroiIJC7Ig1mly/S3bMBxYK1G5CJS4RIX5NG0aUSpFH7rWqY3ZWndqJkrIlLZEhfkuC7h\nzFm4LS3Mmpqlu69Ih67wFJEKlrwgJ55L7q5fx5zGeBFmnfAUkUqWyCAPZsUXBe3n9gLQsmmMCzqL\niEwCiQzysHTCc25eUxBFRBIa5PEUxOlb1+OgIBeRypbIIC+8bjEANbfdQnNjteaSi0hFS2SQF488\nmsLhR5L+0x0clOqjs6dAR49mrohIZUpkkAP0ffAjOEHAqSvuBtAtbUWkYiU2yHPnvIeoqooj/nIL\nRBGvbOja000SEdkjEhvkUWMTube+g9q1qzls7TM8/tzGPd0kEZE9IrFBDtD3ofMAOOeFe1n18hat\n4SkiFSnRQV448WSK++3PwhX3ke3t4tFVG/Z0k0REdrtEBzmOQ98HPoyfz/H6p+9m6cr1e7pFIiK7\nXbKDHOj70PmEdfW8/9HfsvaFFrZ05vZ0k0REdqvEB3k0dSo9n/kHaro7OOeR/8dSlVdEpMIkPsgB\nei/6NIWZs3jH47fy3F9X7OnmiIjsVv5oNjLGXAUsAiLgEmvt0rLnPg5cAATAk8DF1trdu9JDdTW9\nl3+J+s98ipNvupZN553J1Iaq3doEEZE9ZacjcmPMacACa+1i4sC+uuy5LPB+4BRr7UnAwcDiCWrr\nDuXe83627m848+l7WPW7u/dEE0RkL+Zs2ED229+g5ptfw12/blSv8Z61ZH51I6n77sV95WUIgtF/\nYLEIvb2wG1YwG82I/CzgZgBr7UpjTJMxpt5a22Gt7Sk93x/qDcDofkLjzfPI/cs3cT90Lmd/8eO0\nz7ie9BvO3iNNEZHx4y99mJpvfZ3eCz5B/q1v3+XXu+taqb7m+1Rffx1Ob7yGQfW1P6T3ox8nd857\ncDZuxF3XitPdRVTfQNTQiNvaQtX/vZHUskeHvFdUXU3h+EXkTzmVwoknE86eQ9g0BRyH9D13k7n1\nZtJ//hPO1q04YRi/JpMh3GcG4T4z6P7cZRTOOOu1/1CGcXa23qUx5lrgNmvt70rf3w9cYK19tmyb\ny4BLgO9Za7+zo/crFoPI973X3PDtefKK73Lot7+AR4R7zX/AJz85YZ8lslsFAbS1QW1t/N9Ey+fh\nu9+FOXPgfe+DqlGUK6MIenoGR6FVVeCPMF7sH9k6zuB/I7nhBrjwwrgtrgs/+hF8/ONDt1m/HpYt\ngyeegMZGOO44OPJIWLoUrrkGfvvbeHQ8dy5cdhl4HnzjG/Dqqzvui+vCG94Ab3tb/BnPPw8rVsBT\nT428bSm4mTMH5s+HTAZSKdi8GVpb49/dN78Jl16648/dvu38kMYW5A8AHysP8tLj1cDtwBettQ9u\n7/3a2jrHfJzR3FxHW1vnDrcJwpAbvnIdF13/FRp6O8i95e10f/5ygsMOH+vH7nGj6fdkM2n73NdH\n1W//l8IJiwgWHLTN00P6HQSk//wnqq6/Dv/xx3A3tuGEIWFtHd1f+hp9530sDpByUQSFQhyMqdTo\n25XPQzo9pJ31F36EzJ13ABBOnUrvhz9K/vVvJJwzl3CfGdt8tr9sKXWf+yz+04MTDsKGRvre97f0\nnXcBwZy5ZG69merrryP1yEODL8xkKBx9DIXXLaaw8Fii6niHkb7vL2R/eDVhfQPdl3+Rmn/7Nu6m\nTXRd8RWCw48g/cc/kL77T3ivvLxNdyLXHRgRFw8+hN6Pf4q+931gsI99fVT94gb8Z54mnDmTcOYs\nwtpa3I4OnPZ2SPnk3v4uwpmztnlvZ8MG0kvux1+2FLetDXfTRpyuLgqLTiT39ndSXHjsyDumKBry\n+K7+jTc3172mIP8q0Gqt/XHp+xeBo6y1ncaYKcDh1tr7Ss/9E4C19l+3934THeQAT7+0mRuu/SNf\nuvtq9l39NAC5t7+L4oEH4q1ZjffqqxQWn0T35y8f+se7l5qsoZa6989k/vB7uv/5CqIpU4c8tzv7\n7D1rIZ8nOPyI1/Q+qfvupeoXN1A84ihy73gX4dx5Q5//813UXXYp3uqXiLJZOr93Dbl3nQOAu34d\n1T/+IdlN6+mLXKJUivRf7sV7eTUAwbz5hDNmEjZPJ/XAfbjtW8kvOpHeiz5N6onHSD3wF/yVzwyU\nDqKqKnrPu4CeSy4lmjZt+40OQ6p/+ANqvvUvFI49nt6//yz5xSfTcP4HSd93D/nTz6R4+JFU3fg/\nuFu2DLwsSqUoHnIYhTPOIn/aGaTvuI3qn/wIJ4rIn3QKUXU1AP7yJ/E2xBfqRdkanJ74LqWFY48j\nqs4CkO5sJ1qxYiB4yxX3P4COn/8vwYEL8J5/job3vBNv7eBIOmxspHDCIopHLaR4xFE4W7eQeuIx\n/OVPEsyZQ9/5F1JYfNL2R/x70O4O8hOBr1lrX2+MOQa42lp7cum5fYC/Akdaa7uMMb8BbugfvY9k\ndwQ5wNW/Wc4Tz7XxpRkbWfjLH5J64vFttiksPIaOn/wP4bz5Y2qP9+LzpO75M33v/yDU1Ox0e//h\nh8j++Bpyb3kbuXPeO+o/rskY5O6a1TSdcRJuV2f8j/UXvybY/8CB55un1dK2cft3tHRb1pJ6aAnF\noxcOeV0/Z+sWMrfcTOaWm4kyaYrHHk/hmOMIDjmUcFozeB7eUyuoufI7ZG67hchx6LnkUno+f/ng\nSDaK4kPynYxsnY0bqf3KF6j69a+GPF44eiHhrDlEVRnczZtJ3/tnIs8j9zfvIX3773G7u+j5xKfB\n86n+6bU4fX1DXh9ls/Sd8176zr+A4hFHDfZ9/TpqL/scmdtuGdzW9ykefChRbS1kMnjPP4e39lWi\nbA29H/ko4Zw5RKk0UXU1gTmY4iGH4XR1Uff3nyBz958Ia+twu+K/sbCmFre7i9yb3krHT34Wlwh6\ne8ncejP+ymdwX30F75U1+E+twMkP3t+oeMCBdF15NYUTTy77IRRI/+H3VP/sv/FeeZncu86h90Pn\nEc7fd2CT5uY6Nr7Ugv/oUvynVkBYKrtUV9P3nvcTNTYN9n3tq9R++QsE8+aTf8ObKBz/upFLNwmw\nW4McwBjzbeBUIAQuBhYC7dbam4wx55ceKxJPP/zUjqYf7q4gX7+5hy/99yOkfZcvn3css55djhMU\nCebvS9TQQO1ln6Pq178ibGgk9+5zcDdtwt2wniidJlhwEMWDDqZ46OEUj144Ym0wteQB6s/7AG77\nVoL5+9L5vWsonHTKiG1xNm2i5utfpvoXNww8lj/pFLq+fSWBOfi19zuKIJcbXQ3ztSgWqfrlz8l+\n/9+JqjLkzziLwulnkj/xFCiNwEb7Po3vfDOppQ+TP/Ns0n++i7Cxka5v/ive88+SueN2/LWv0nnF\nV+n76IUDL3M62qn62X+Tue0WUo8/NvB44bgT6DvnvZBK4b34At6zq0jfd++QkCkX+T5h83S81pb4\n9ccci7txE97LqyksPIbeT/4dqQcfIH3XH3FbWwhnzyHY/0DCadNw2zbgrmvF3bKFqKqKqLoad/16\n3I52CkceTfdXv4H30otkbrmJ1AP34ZTNcigc/zo6v/PvBIcfgfespf78D+A//xwAwazZ9Fz6z9S9\n991sat2Mk88TzpxJVFe/3R9j+o9/wH98GYUTFsWrZpUPJnI5qn7+P2Sv+u7AiHjIz8DziKqqcbu7\nyJ95Nh0/+DFu2way//E9Mjf9htw73kXnf1y7451YdzfpJfeTuu9ewuZ96L3oU2P6G5yMA5XR2O1B\nPp52V5AD3L+8hetuX8Xc6bV84cPHkkmVnWSNIjK/upG6yy4dPBwtq6sNbJZOUyzV7/InnUzxhEWk\n//gH6i75NIQhube/k8zvbsIJQ/refQ5EEd5LL8WHf45D5Pu47e04Pd0UDzuC7s9fTtWvfk7mjtuJ\nfJ++v/0wvRf//cCo0mnfSur++4iy1RQXHkvUNGWH/fYfe5S6Sy/Bf3oFYWMj4cxZFI88mq6vf2vI\nSIYgiEsIvh8f9kYR/rOr8J5+Cm/NGsI5cygedDDBvPl4a1bjP/UkvrWEU6YQ7Ls/0ZQpVF/7Q/yV\nzxBl40Nip6cHgHBaMz2f/Dv6PnYhUW0d5PP4q57BbdswcNIrbJ4ejypdl+yV36HmO9+k711/Q+eP\nr4t/D5d+BqdYjH/mmQxOJgMdHfR+5GN0/Z9/JfP731Hz5S/gbVhP5PsUTjyF/Cmnkn7wflJ/uQdn\n2N9x8eBD6Dv3/eTOfS+RnyL12KOkli3FXf0SXsta3NYWgrnz6LnkUgpnnIXT1Unt5Z+n6n9/OfAe\nYVMTwUEH4768ZiD0AcIpUwinTMXJ53F6eog8j96/u4TeCz4xdHSYy+H0dOPkclAsEs6eM+QozOns\nIPvdbxPMm0ffh86HqqrxD7WeHlIPPYjT24dTyON0dOA/8xT+iuW4LWvp/dhF9F78maH17p6eeMe8\nm8oRCvJRb1+ZQQ5w/R2ruPeJFhYftg8Xvu1QnGF/nM6GDXjrWwmmz4hriX19+M8/i2dX4S9/gtTD\nD+E/tXxgZBV5Hk4QENbV03Hdzymcenocpp+9GH/VynibqiqCWbPBdXGKRSLPo+/8C4b8Q0//8Q/U\nfPly/JdeJHIc8m98C053N6m/PjAQaBAfrvpnnE77iadTOPU0ovoGCEOczZvJfv9Kqn/ynzhhSOH4\n1+F0tOO2tOB2dlA84EA6bvxfgv0PxFu1krrPfHLE8tKuiEo3Keu57IuEjU2klj5M+s47qLrxetzO\nDsLGRoJ5++KvfBqnUNjm9eG0ZvKnnkbmdzcRzpjJlnseHNjZ+A8/ROb2Wykc/zryp59Js5un+Na3\nxzuoadNwN24kqqqi57Ofo/djHx96uN3aQvqO24myWYL9DiDYb3+i5uYx9TF9x+34Ty0nf8rpFI87\nPp7hANDVhbtlM2Hz9Ak98qnEUKvEPoOCfJdeUyiGfOcXj/FiSwfnnn4Ab1k0hnp4VxepRx8hteQB\n0g/eD2FI57//gOCQQwe3yefj0Jm+T3yme/hsgpEEAenbbiH7g++RejIO2cLRC8m/4c1QLJBa9ij+\nY8twOzuAuCQQNTbibNkysGMp7n8AXf/+g8G6ZBhS882vkf3BVYSNjfS97wNUX/dfOPk8uTe9lbB5\nOk5vD4QBwYEHUTz0cIJ998Nb+wqetXgvryaYO5/iEUcSHHwIztatcbli7SvkTzxlxNk/TvtWqn/6\nE6p/fA1OVxfFww6neORCgrlz6Z8x5b3wHJm77sRt20DkOLTfdNvQWuowzc11tK1eR91nL6bqd/+P\n3Fmvp+tb/0a4736j+IUlVyWGWiX2GRTku/y6zR19fP36R2nvynP2cXN4/5kLcN296Cx2FOE9/RTR\n1KnbTncKAprXWLp/czPpe+7Gad9KNGUq4ZSpFBceQ8+n/n7EEWLmlz+n7nOX4BQKBNP3oevKq8m/\n8c0T248giOfSbq+uGob4Tz4OQUDxuBN2+FYDv+sowl376jZlicmqEkOtEvsMCvIxvXbj1l6u+vWT\ntG7qYeGCaVz0jsOG1sz3YmPtt//Iw6TvuYveiz5F1DRlAlo2cfSPu3JUYp9hfIN8Utz9cDSmNVZz\nxYeP5ZD5TTz+3Ea+feNjbO7o2/kLE6x4wuvo+ecrEhfiIrJrKibIAbJVKf7hvUdx8pEzWbOuk3/5\n2VLsy1t2/kIRkb1YRQU5gO+5fPTNB/PB1x9Ed1+Rf/vVE9z219XkCrtwVzMRkb1IxQU5gOM4nHXs\nHD73/qPJVvn89i8v8k//uYTbH1pDb6648zcQEdmLVGSQ9zPzmvjmxxfx9hP3pRhE/ObeF/jnH/2V\nu5e9SjHY9r4PIiJ7o4oOcoDa6hTvPnV/vvupxbzrlP0oBiE3/ulZvvhfD/PwM+sJRriRj4jI3iSZ\nd5uZANmqFO84aT9OXzibWx9Yzb1PrOXHtzzNb+6t4uzj5nDKkbPIVunHJSJ7HyXTMPXZNB98w0Gc\nffwc7nzkFR5c0cr//fPz3HTfixx54DROOHg6Rx4wlXRC5qCLyOSnIN+OfZqyfPiNhnefuj9/eWIt\nD6xYx6OrNvDoqg1kUh5HHTiV48x0jjhgamIuLBKRyUlBvhO11Sneunhf3rJoPq9s6OKRlRtYumo9\nj6zcwCMrN5D2XQ6e38RRB0zliAOmMq1hF27nKiIyDhTko+Q4DvP2qWPePnWcc9r+vLy+i0ftBh57\nto3lL2xi+QubAJgxJcvh+03h0P2msP+seuqze/8KRCKSbAryMXAch/kz6pg/o45zTjuAjVt7WfFi\nHOarXt7KXcte5a5l8XJUTXUZ9p1Rx6H7TuGI/acwvSm7h1svIpONgnwcTGus5oxj5nDGMXMoBiEv\nrG1n5ZotrF7XyZp1nTz+3EYef24jAPs0VbNgbiP7z6xnv5n1zJpWQ8qv+FmgIvIaKMjHme+5mHlN\nmHmDCx9sbO/lqRc3s+LFTTyzZgsPLG/lgeWtQHxn1umN1cycWsM+U6ppboz/mzEly9SGKtwKuHWr\niLw2CvLdYFpDNacvnM3pC2cThCGtG3t4sbWD1es6aWnromVTD088v3Gb12XSHrOn1XDg3Caa6zPM\n26eWmVNrqKnyt1npSEQql4J8N/NclznTa5kzvZZTBxdGp6MnT9uWXtq29rJhay+tm3pY29bFmnWd\nvNjSMeQ9MimPKfUZmhurmTk1y6ypNUxrqKIq41OV9qjLpqmt3vHK7yIyeSjI9xL12TT12TQHzG4Y\n8ngxCMlFDk+uWsfL67vYsKWXzR19bOroo3VTz8BsmeGa6jLMnV7L7Gk1NNVlaKrL0FiXYUpdFQ21\naZVsRCYRBflezvdcZjbXUeM7nDhsucyu3gKtm7pp3dTD5o4++vIBuULAls4cr2zoGjItspznOjTW\nxuHeUJumoSaN77m4roPvOezTlGXu9FpmTavB93QiVmRvpyBPsNrqFAvmNLJgTuOIz3f25Gnd1MPW\nrhxbO3Ns6cqxuSM3MKJ/saUKQvanAAAKDElEQVSDcAdL/cWBn6a+JkNDTZr6mhS11XHZpi6boi4b\nP1ad8Un7HpmUSybt4Y1m4WkRGTcK8kmsLpumbgcXJIVhRGdvgY7uPEEYEoZQKAa0bOzm5Q1dvNrW\nxdbOHK9s6OSlYPRLraZ9l6qMT02VT302TV02RUNthqn1VUypj3cKKd8rbefRVJdR+Iu8BgryCua6\nDg01cWmlXPnUSYAoiujuK9LZk6ert0BnT4Gu0g6gs6dAb75IvhCQL4T05Yv05gP6ckU6ewqs29TD\nznYBnuswpT5DU10V6ZRLynOpq81AGFGd8ahO+2SrfGqqU9RUpahKe2RSXunEbnxEoFk8UskU5LJT\njuNQW50a00yYIAzp6i2ytXOwpNPZU6BQDCkUQ7r7Cmxs76Ntay/PvrJ1TO1L+y71NWkaazM01qZp\nqM1QlfZIeS4p38X3XTIpj5Qf7yR8z8X3HKoyPvU1aRqyaTLp+MZnUanUpB2DJImCXCaU57oDo/75\nM+p2uG0YRhSCOODr6qtpWddOby6gN1+kp69IV2+B7t4CuUJ8UrcvH9DRnae9O097V26nNf8dcWDg\nyKH/3EBDbYb6bJrqjE8241OV8UinPDK+Szrlka2KH6/O+PFOwndJ+x6+5+D7Lr7rkkq5miEkE05B\nLnsN13XIuHHZZFpjNVFh19ZPDcOIzp442PvyAcXSTqH/v1wxoFgMKQYRxSCkJ1ekoztPR3eevkKA\n6zi4DuQKIe3dOdas6yQIx7ZjKJdOxUcE1ek49KszHlXpeM5/OhUfOThuvAOZ2pjFJaIumyZb5ZNJ\nxT8P33NwXQfHceJlvRxwiB/zSrONUr6n2z1UKAW5TBqu69BQm6GhNjMu7xdGEb25Yum/gN5ckUIx\nJF86IujNFenJxUcLhWJIIQjJF0KCMBzyfa4QkMvHRxbt3XlyhWBc2jeS/vMG/TuCmqoU1WmPIIwo\nBhFRFJHy49lF8Q7CLe0IXGqqfRpq4hPkKd+NdxoOpVJU/xGHO/Cc7D0U5CLb4ToONVXxCdbxFIQh\nuXxYKg8VCYKIIIwIo4h0VYpXWtvjk8i5IrlCQD4f7xSiKCKK4h1MXEGKCCMIgvgoI1cI4pPQPflx\nO5oYiePEVxd7rkMUxSUp33MGykzVpSuMM+l4RxGW+uaWzrXUVA/uXIIwIptNExaDgdenUx7pVFym\n8kpHHK7rEIYRxTAiDKPSEUipxJXxK/56h1EFuTHmKmAR8e/sEmvt0rLnzgC+BQSABS601mrFYpHt\n8FyXbJVbWgN26NFDc3Mds5te++IkURSRL8Qnk3vzAb7r4HkOruOQL4bkShePBUFIUDo3UT4TKd5x\nxOWqIIx3FOVHI7l8QBBFxONyh2IQ0psrsrkzR6G4+//5ZzM+ddkU6ZSH6zg4pSOJ/nMX5ecpHAe8\n0glv13EGdoyOw2DJq/Qax4mP9Pqny/bvYPr/n+r/3ncHdjhe//ap+LHdcfSy0yA3xpwGLLDWLjbG\nHAL8FFhctsm1wBnW2leNMb8G3gTcPiGtFZFRcRwnLp+kd/8yhMWgdLSRi89T9AdcEEZ09xXo6inQ\nlw/i0bbn0thYzboNnfT0xaWqfCEgXwwpFEKCKCIIQsIwikPSc/Ech2IYl63yxYDu0pTYzp48HT2F\nOJhLo/2JOioZLddxcEsnNTzX4T1nHMCZx8wZ988ZzYj8LOBmAGvtSmNMkzGm3lrbfyenY8u+bgOm\njnsrRSQx+mvqI5Wkmtn2aKO5uY62qROz4Er/+Yqw7CAhjEohH4QDJR/HiUs3faVyV75QKmURH5Xk\nCyGFYnw0ki+dPO/f4eQLQekzotKOp3T0UoyvrSgf8TeN0/mb4UYT5DOAZWXft5Ue6wDoD3FjzEzg\nDcCXdvRmTU1ZfH/so4Tm5h1PYZusKrHfldhnqMx+V2KfYfz6PZaTndsUfIwx04FbgU9ba0e+HV/J\nli09Y/jIWHNzHW1tnWN+fVJVYr8rsc9Qmf2uxD7Drvd7R6E/miBvIR6B95sFtPZ/Y4ypB/4AXGGt\nvXPUrRIRkXExmjk7dwLnAhhjjgFarLXlu5ErgaustXdMQPtERGQndjoit9YuMcYsM8YsAULgYmPM\n+UA78EfgI8ACY8yFpZf8wlp77UQ1WEREhhpVjdxae9mwh54s+3piTsOKiMioVPblUCIik4CCXEQk\n4RTkIiIJ50RjvH+ziIjsHTQiFxFJOAW5iEjCKchFRBJOQS4iknAKchGRhFOQi4gknIJcRCThErP4\n8o7WDZ1sjDH/CpxC/Pv5FrAUuAHwiG8h/GFrbW7PtXBiGGOqgaeArwN3Uxl9/iDwT0AR+DKwnEnc\nb2NMLXA90ER8n6avAeuA/yT+t73cWvupPdfC8WWMORz4HfEdYv/DGDOXEX6/pb+DzxLfmPBaa+1/\n78rnJGJEXr5uKHABcPUebtKEKS1mfXipr28Cvgf8C3CNtfYU4HngY3uwiRPpi8Dm0teTvs/GmKnA\nV4CTgbcB72Ty9/t8wFprzyC+Pfb3if/GL7HWngQ0GGPevAfbN26MMTXAD4gHJf22+f2WtvsycDZw\nOvAPxpgpu/JZiQhyhq0bCjSVFrSYjO4D3lP6eitQQ/zLvaX02K3Ev/BJxRhzMHAocFvpodOZ5H0m\n7tNd1tpOa22rtfYiJn+/NzK4rm8T8Y57v7Ij7MnU5xzwFuLFefqdzra/39cBS6217dbaXuBB4KRd\n+aCkBPkM4rVC+/WvGzrpWGsDa2136dsLgNuBmrLD6w3AzD3SuIl1JfCPZd9XQp/3BbLGmFuMMfcb\nY85ikvfbWvsrYJ4x5nniQcvngC1lm0yaPltri6VgLjfS73d4vu3yzyApQT7cNuuGTjbGmHcSB/nf\nDXtq0vXdGPMR4K/W2pe2s8mk63OJQzw6/RviksN1DO3rpOu3MeZDwMvW2gOBM4GfD9tk0vV5B7bX\n113+GSQlyHe4buhkY4x5I3AF8GZrbTvQVToRCDCboYdqk8FbgXcaYx4CLgS+xOTvM8B6YElp5PYC\n0Al0TvJ+n0S8shjW2ieBamBa2fOTsc/lRvq7Hp5vu/wzSEqQ72zd0EnDGNMAfBd4m7W2/8TfXcA5\npa/PASbV+qjW2vdZa4+31i4C/ot41sqk7nPJncCZxhi3dOKzlsnf7+eJa8IYY+YT77xWGmNOLj3/\nN0y+Ppcb6ff7MHC8MaaxNKvnJOD+XXnTxNzG1hjzbeBUSuuGlvbmk44x5iLgq8CzZQ+fRxxwVcAa\n4KPW2sLub93EM8Z8FVhNPGq7nkneZ2PMJ4hLaADfIJ5qOmn7XQqqnwL7EE+v/RLx9MMfEw8sH7bW\n/uP23yE5jDHHEp/72RcoAGuBDwI/Y9jv1xhzLvB54imYP7DW3rgrn5WYIBcRkZElpbQiIiLboSAX\nEUk4BbmISMIpyEVEEk5BLiKScApyEZGEU5CLiCTc/wc11nIwNRDsJgAAAABJRU5ErkJggg==\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "metadata": {
        "id": "4RIGVSojmy-R",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "### Predict"
      ]
    },
    {
      "metadata": {
        "id": "tnigE74rm39a",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "unlabeled_images_test = pd.read_csv('gdrive/My Drive/dataML/test.csv')\n",
        "#unlabeled_images_test = pd.read_csv('test.csv')"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "3Fotf1KrpXVE",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "X_unlabeled = unlabeled_images_test.values.reshape(unlabeled_images_test.shape[0],28,28,1)/255"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "jeRnCHzdutQ0",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "y_pred = model.predict(X_unlabeled)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "3M_fePteu-3X",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "y_label = np.argmax(y_pred, axis=1)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "RX5PuSUmvRri",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "### Save csv"
      ]
    },
    {
      "metadata": {
        "id": "zU5Q1fSRvVbn",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "imageId = np.arange(1,y_label.shape[0]+1).tolist()\n",
        "prediction_pd = pd.DataFrame({'ImageId':imageId, 'Label':y_label})\n",
        "prediction_pd.to_csv('gdrive/My Drive/dataML/out_cnn05.csv',sep = ',', index = False)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "_qetEX7AgBQY",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "# Tensorflow"
      ]
    },
    {
      "metadata": {
        "id": "4keUL7d0gBQZ",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "### Helper functions for batch learning"
      ]
    },
    {
      "metadata": {
        "id": "czdbjPfcgBQd",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "def one_hot_encode(vec, vals=10):\n",
        "    '''\n",
        "    For use to one-hot encode the 10- possible labels\n",
        "    '''\n",
        "    n = len(vec)\n",
        "    out = np.zeros((n, vals))\n",
        "    out[range(n), vec] = 1\n",
        "    return out"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "Afc2XmK2gBQh",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "class CifarHelper():\n",
        "    \n",
        "    def __init__(self):\n",
        "        self.i = 0\n",
        "        \n",
        "        # Intialize some empty variables for later on\n",
        "        self.training_images = None\n",
        "        self.training_labels = None\n",
        "        \n",
        "        self.test_images = None\n",
        "        self.test_labels = None\n",
        "    \n",
        "    def set_up_images(self):\n",
        "        \n",
        "        print(\"Setting Up Training Images and Labels\")\n",
        "        \n",
        "        # Vertically stacks the training images\n",
        "        self.training_images = train_images.as_matrix()\n",
        "        train_len = self.training_images.shape[0]\n",
        "        \n",
        "        # Reshapes and normalizes training images\n",
        "        self.training_images = self.training_images.reshape(train_len,28,28,1)/255\n",
        "        # One hot Encodes the training labels (e.g. [0,0,0,1,0,0,0,0,0,0])\n",
        "        self.training_labels = one_hot_encode(train_labels.as_matrix().reshape(-1), 10)\n",
        "        \n",
        "        print(\"Setting Up Test Images and Labels\")\n",
        "        \n",
        "        # Vertically stacks the test images\n",
        "        self.test_images = test_images.as_matrix()\n",
        "        test_len = self.test_images.shape[0]\n",
        "        \n",
        "        # Reshapes and normalizes test images\n",
        "        self.test_images = self.test_images.reshape(test_len,28,28,1)/255\n",
        "        # One hot Encodes the test labels (e.g. [0,0,0,1,0,0,0,0,0,0])\n",
        "        self.test_labels = one_hot_encode(test_labels.as_matrix().reshape(-1), 10)\n",
        "\n",
        "        \n",
        "    def next_batch(self, batch_size):\n",
        "        # Note that the 100 dimension in the reshape call is set by an assumed batch size of 100\n",
        "        x = self.training_images[self.i:self.i+batch_size]\n",
        "        y = self.training_labels[self.i:self.i+batch_size]\n",
        "        self.i = (self.i + batch_size) % len(self.training_images)\n",
        "        return x, y"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "1qAiPT1-gBQp",
        "colab_type": "code",
        "outputId": "77cb7989-501e-41d6-84a5-e65cef94fd1f",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 51
        }
      },
      "cell_type": "code",
      "source": [
        "# Before Your tf.Session run these two lines\n",
        "ch = CifarHelper()\n",
        "ch.set_up_images()\n",
        "\n",
        "# During your session to grab the next batch use this line\n",
        "# (Just like we did for mnist.train.next_batch)\n",
        "# batch = ch.next_batch(100)"
      ],
      "execution_count": 61,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Setting Up Training Images and Labels\n",
            "Setting Up Test Images and Labels\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "QUkUv8sKgBQw",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Creating the Model\n",
        "\n"
      ]
    },
    {
      "metadata": {
        "id": "l1XSMzIpgBQx",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "** Create 2 placeholders, x and y_true. Their shapes should be: **\n",
        "\n",
        "* X shape = [None,28,28,1]\n",
        "* Y_true shape = [None,10]\n",
        "\n",
        "** Create three more placeholders \n",
        "* lr: learning rate\n",
        "* step：for learning rate decay\n",
        "* drop_rate"
      ]
    },
    {
      "metadata": {
        "id": "8Y_4DDQvgBQz",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "X = tf.placeholder(tf.float32, shape=[None,28,28,1])\n",
        "Y_true = tf.placeholder(tf.float32, shape=[None,10])\n",
        "\n",
        "lr = tf.placeholder(tf.float32)\n",
        "step = tf.placeholder(tf.int32)\n",
        "drop_rate = tf.placeholder(tf.float32)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "5egQVsS4NhxW",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "### Initialize Weights and bias\n",
        "Five layers with 200, 100, 60, 30 and 10 neurons"
      ]
    },
    {
      "metadata": {
        "id": "RQ_UDA08dBcN",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "L = 200\n",
        "M = 100\n",
        "N = 60\n",
        "O = 30"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "xiVQdlDsN1W-",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "W1 = tf.Variable(tf.truncated_normal([784, L], stddev=0.1))  # 784 = 28 * 28\n",
        "B1 = tf.Variable(tf.ones([L])/10)\n",
        "\n",
        "W2 = tf.Variable(tf.truncated_normal([L, M], stddev=0.1))\n",
        "B2 = tf.Variable(tf.ones([M])/10)\n",
        "\n",
        "W3 = tf.Variable(tf.truncated_normal([M, N], stddev=0.1))\n",
        "B3 = tf.Variable(tf.ones([N])/10)\n",
        "\n",
        "W4 = tf.Variable(tf.truncated_normal([N, O], stddev=0.1))\n",
        "B4 = tf.Variable(tf.ones([O])/10)\n",
        "\n",
        "W5 = tf.Variable(tf.truncated_normal([O, 10], stddev=0.1))\n",
        "\n",
        "B5 = tf.Variable(tf.zeros([10]))"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "u4RDJq8pO_Og",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "### layers"
      ]
    },
    {
      "metadata": {
        "id": "x52yDbi6bCMb",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "XX = tf.reshape(X,[-1,784])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "DM5_O098O4Di",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "Y1 = tf.nn.relu(tf.matmul(XX, W1) + B1)\n",
        "Y1d = tf.nn.dropout(Y1,rate = drop_rate)\n",
        "\n",
        "Y2 = tf.nn.relu(tf.matmul(Y1, W2) + B2)\n",
        "Y2d = tf.nn.dropout(Y2,rate = drop_rate)\n",
        "\n",
        "Y3 = tf.nn.relu(tf.matmul(Y2, W3) + B3)\n",
        "Y3d = tf.nn.dropout(Y3,rate = drop_rate)\n",
        "\n",
        "Y4 = tf.nn.relu(tf.matmul(Y3, W4) + B4)\n",
        "Y4d = tf.nn.dropout(Y4,rate = drop_rate)\n",
        "\n",
        "Ylogits = tf.matmul(Y4d, W5) + B5\n",
        "Y = tf.nn.softmax(Ylogits)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "tT4TvNz-gBRI",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "### Loss Function"
      ]
    },
    {
      "metadata": {
        "id": "wkqQ5GurgBRJ",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "#cross_entropy = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits_v2(labels=Y_true,logits=Ylogits))\n",
        "cross_entropy = tf.losses.softmax_cross_entropy(onehot_labels = Y_true, logits = Ylogits)\n",
        "#cross_entropy = -tf.reduce_mean(y_true * tf.log(Ylogits)) * 1000.0 "
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "jmnEUVWxgBRM",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "### Optimizer"
      ]
    },
    {
      "metadata": {
        "id": "MoyIlzCagBRN",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "lr = 0.0001 + tf.train.exponential_decay(learning_rate = 0.003, \n",
        "                                         global_step = step,\n",
        "                                         decay_steps = 2000,\n",
        "                                         decay_rate = 1/math.e\n",
        "                                        )\n",
        "\n",
        "#optimizer = tf.train.GradientDescentOptimizer(learning_rate = 0.005)\n",
        "optimizer = tf.train.AdamOptimizer(learning_rate=lr)\n",
        "train = optimizer.minimize(cross_entropy)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "47rAzeVNgBRP",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "### Intialize Variables"
      ]
    },
    {
      "metadata": {
        "id": "7WG1AszIgBRQ",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "init = tf.global_variables_initializer()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "rseSYLjggBRb",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "### Saving the Model"
      ]
    },
    {
      "metadata": {
        "id": "OwdUgOG4gBRc",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "saver = tf.train.Saver()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "xGdHTpE0gBRe",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Graph Session\n",
        "\n",
        "** Perform the training and test print outs in a Tf session and run your model! **"
      ]
    },
    {
      "metadata": {
        "id": "pQHEYbyZgBRf",
        "colab_type": "code",
        "outputId": "12d77dbd-681f-47c3-cceb-db6c403aeca9",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 10237
        }
      },
      "cell_type": "code",
      "source": [
        "history = {'acc_train':list(),'acc_val':list(),\n",
        "           'loss_train':list(),'loss_val':list(),\n",
        "          'learning_rate':list()}\n",
        "with tf.Session() as sess:\n",
        "    sess.run(tf.global_variables_initializer())\n",
        "    \n",
        "    for i in range(20000):\n",
        "        batch = ch.next_batch(100)\n",
        "        sess.run(train, feed_dict={X: batch[0], Y_true: batch[1], step: i, drop_rate: 0.25})\n",
        "        \n",
        "        # PRINT OUT A MESSAGE EVERY 100 STEPS\n",
        "        if i%100 == 0:\n",
        "            \n",
        "            # Test the Train Model\n",
        "            feed_dict_train = {X: batch[0], Y_true: batch[1], drop_rate: 0.25}\n",
        "            feed_dict_val = {X:ch.test_images, Y_true:ch.test_labels, drop_rate: 0}\n",
        "\n",
        "            matches = tf.equal(tf.argmax(Y,1),tf.argmax(Y_true,1))\n",
        "            acc = tf.reduce_mean(tf.cast(matches,tf.float32))\n",
        "            history['acc_train'].append(sess.run(acc, feed_dict = feed_dict_train))\n",
        "            history['acc_val'].append(sess.run(acc, feed_dict = feed_dict_val))\n",
        "\n",
        "            history['loss_train'].append(sess.run(cross_entropy, feed_dict = feed_dict_train))\n",
        "            history['loss_val'].append(sess.run(cross_entropy, feed_dict = feed_dict_val))\n",
        "            \n",
        "            history['learning_rate'].append(sess.run(lr, feed_dict = {step: i}))\n",
        "            print(\"Iteration {}:\\tlearning_rate={:.6f},\\tloss_train={:.6f},\\tloss_val={:.6f},\\tacc_train={:.6f},\\tacc_val={:.6f}\"\n",
        "                  .format(i,history['learning_rate'][-1],\n",
        "                          history['loss_train'][-1],\n",
        "                          history['loss_val'][-1],\n",
        "                          history['acc_train'][-1],\n",
        "                          history['acc_val'][-1]))\n",
        "            \n",
        "            print('\\n')\n",
        "        \n",
        "    saver.save(sess,'models_saving/my_model.ckpt')"
      ],
      "execution_count": 71,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Iteration 0:\tlearning_rate=0.003100,\tloss_train=2.213235,\tloss_val=2.256213,\tacc_train=0.270000,\tacc_val=0.202381\n",
            "\n",
            "\n",
            "Iteration 100:\tlearning_rate=0.002954,\tloss_train=0.390136,\tloss_val=0.342713,\tacc_train=0.910000,\tacc_val=0.923810\n",
            "\n",
            "\n",
            "Iteration 200:\tlearning_rate=0.002815,\tloss_train=0.257970,\tloss_val=0.229378,\tacc_train=0.930000,\tacc_val=0.938095\n",
            "\n",
            "\n",
            "Iteration 300:\tlearning_rate=0.002682,\tloss_train=0.292719,\tloss_val=0.179952,\tacc_train=0.920000,\tacc_val=0.957143\n",
            "\n",
            "\n",
            "Iteration 400:\tlearning_rate=0.002556,\tloss_train=0.140316,\tloss_val=0.173160,\tacc_train=0.950000,\tacc_val=0.957143\n",
            "\n",
            "\n",
            "Iteration 500:\tlearning_rate=0.002436,\tloss_train=0.174110,\tloss_val=0.205662,\tacc_train=0.960000,\tacc_val=0.959524\n",
            "\n",
            "\n",
            "Iteration 600:\tlearning_rate=0.002322,\tloss_train=0.097515,\tloss_val=0.151778,\tacc_train=0.960000,\tacc_val=0.961905\n",
            "\n",
            "\n",
            "Iteration 700:\tlearning_rate=0.002214,\tloss_train=0.064677,\tloss_val=0.149972,\tacc_train=0.980000,\tacc_val=0.961905\n",
            "\n",
            "\n",
            "Iteration 800:\tlearning_rate=0.002111,\tloss_train=0.111690,\tloss_val=0.178004,\tacc_train=0.960000,\tacc_val=0.957143\n",
            "\n",
            "\n",
            "Iteration 900:\tlearning_rate=0.002013,\tloss_train=0.021550,\tloss_val=0.174571,\tacc_train=1.000000,\tacc_val=0.959524\n",
            "\n",
            "\n",
            "Iteration 1000:\tlearning_rate=0.001920,\tloss_train=0.205623,\tloss_val=0.152802,\tacc_train=0.960000,\tacc_val=0.971429\n",
            "\n",
            "\n",
            "Iteration 1100:\tlearning_rate=0.001831,\tloss_train=0.094505,\tloss_val=0.165903,\tacc_train=0.960000,\tacc_val=0.964286\n",
            "\n",
            "\n",
            "Iteration 1200:\tlearning_rate=0.001746,\tloss_train=0.078548,\tloss_val=0.144696,\tacc_train=0.960000,\tacc_val=0.966667\n",
            "\n",
            "\n",
            "Iteration 1300:\tlearning_rate=0.001666,\tloss_train=0.087408,\tloss_val=0.187368,\tacc_train=0.970000,\tacc_val=0.954762\n",
            "\n",
            "\n",
            "Iteration 1400:\tlearning_rate=0.001590,\tloss_train=0.110445,\tloss_val=0.134932,\tacc_train=0.950000,\tacc_val=0.971429\n",
            "\n",
            "\n",
            "Iteration 1500:\tlearning_rate=0.001517,\tloss_train=0.097374,\tloss_val=0.154824,\tacc_train=0.990000,\tacc_val=0.964286\n",
            "\n",
            "\n",
            "Iteration 1600:\tlearning_rate=0.001448,\tloss_train=0.015218,\tloss_val=0.135954,\tacc_train=1.000000,\tacc_val=0.971429\n",
            "\n",
            "\n",
            "Iteration 1700:\tlearning_rate=0.001382,\tloss_train=0.052718,\tloss_val=0.144328,\tacc_train=0.970000,\tacc_val=0.971429\n",
            "\n",
            "\n",
            "Iteration 1800:\tlearning_rate=0.001320,\tloss_train=0.007530,\tloss_val=0.150683,\tacc_train=0.990000,\tacc_val=0.969048\n",
            "\n",
            "\n",
            "Iteration 1900:\tlearning_rate=0.001260,\tloss_train=0.042407,\tloss_val=0.132203,\tacc_train=0.980000,\tacc_val=0.973810\n",
            "\n",
            "\n",
            "Iteration 2000:\tlearning_rate=0.001204,\tloss_train=0.016666,\tloss_val=0.156475,\tacc_train=0.990000,\tacc_val=0.964286\n",
            "\n",
            "\n",
            "Iteration 2100:\tlearning_rate=0.001150,\tloss_train=0.016974,\tloss_val=0.142225,\tacc_train=0.990000,\tacc_val=0.966667\n",
            "\n",
            "\n",
            "Iteration 2200:\tlearning_rate=0.001099,\tloss_train=0.006253,\tloss_val=0.161666,\tacc_train=0.980000,\tacc_val=0.966667\n",
            "\n",
            "\n",
            "Iteration 2300:\tlearning_rate=0.001050,\tloss_train=0.033346,\tloss_val=0.148638,\tacc_train=0.980000,\tacc_val=0.971429\n",
            "\n",
            "\n",
            "Iteration 2400:\tlearning_rate=0.001004,\tloss_train=0.012736,\tloss_val=0.141769,\tacc_train=1.000000,\tacc_val=0.971429\n",
            "\n",
            "\n",
            "Iteration 2500:\tlearning_rate=0.000960,\tloss_train=0.008008,\tloss_val=0.149648,\tacc_train=1.000000,\tacc_val=0.969048\n",
            "\n",
            "\n",
            "Iteration 2600:\tlearning_rate=0.000918,\tloss_train=0.005308,\tloss_val=0.146539,\tacc_train=1.000000,\tacc_val=0.969048\n",
            "\n",
            "\n",
            "Iteration 2700:\tlearning_rate=0.000878,\tloss_train=0.012516,\tloss_val=0.136559,\tacc_train=1.000000,\tacc_val=0.973810\n",
            "\n",
            "\n",
            "Iteration 2800:\tlearning_rate=0.000840,\tloss_train=0.005838,\tloss_val=0.151297,\tacc_train=1.000000,\tacc_val=0.973810\n",
            "\n",
            "\n",
            "Iteration 2900:\tlearning_rate=0.000804,\tloss_train=0.033810,\tloss_val=0.155132,\tacc_train=1.000000,\tacc_val=0.973810\n",
            "\n",
            "\n",
            "Iteration 3000:\tlearning_rate=0.000769,\tloss_train=0.013711,\tloss_val=0.139323,\tacc_train=1.000000,\tacc_val=0.971429\n",
            "\n",
            "\n",
            "Iteration 3100:\tlearning_rate=0.000737,\tloss_train=0.014888,\tloss_val=0.167029,\tacc_train=1.000000,\tacc_val=0.971429\n",
            "\n",
            "\n",
            "Iteration 3200:\tlearning_rate=0.000706,\tloss_train=0.002667,\tloss_val=0.169782,\tacc_train=1.000000,\tacc_val=0.969048\n",
            "\n",
            "\n",
            "Iteration 3300:\tlearning_rate=0.000676,\tloss_train=0.003653,\tloss_val=0.160170,\tacc_train=1.000000,\tacc_val=0.973810\n",
            "\n",
            "\n",
            "Iteration 3400:\tlearning_rate=0.000648,\tloss_train=0.041257,\tloss_val=0.192027,\tacc_train=0.990000,\tacc_val=0.969048\n",
            "\n",
            "\n",
            "Iteration 3500:\tlearning_rate=0.000621,\tloss_train=0.001845,\tloss_val=0.163374,\tacc_train=1.000000,\tacc_val=0.973810\n",
            "\n",
            "\n",
            "Iteration 3600:\tlearning_rate=0.000596,\tloss_train=0.007610,\tloss_val=0.167219,\tacc_train=1.000000,\tacc_val=0.973810\n",
            "\n",
            "\n",
            "Iteration 3700:\tlearning_rate=0.000572,\tloss_train=0.009149,\tloss_val=0.170326,\tacc_train=1.000000,\tacc_val=0.971429\n",
            "\n",
            "\n",
            "Iteration 3800:\tlearning_rate=0.000549,\tloss_train=0.006270,\tloss_val=0.180801,\tacc_train=1.000000,\tacc_val=0.971429\n",
            "\n",
            "\n",
            "Iteration 3900:\tlearning_rate=0.000527,\tloss_train=0.001165,\tloss_val=0.168772,\tacc_train=1.000000,\tacc_val=0.971429\n",
            "\n",
            "\n",
            "Iteration 4000:\tlearning_rate=0.000506,\tloss_train=0.007112,\tloss_val=0.166344,\tacc_train=1.000000,\tacc_val=0.973810\n",
            "\n",
            "\n",
            "Iteration 4100:\tlearning_rate=0.000486,\tloss_train=0.003821,\tloss_val=0.173032,\tacc_train=1.000000,\tacc_val=0.973810\n",
            "\n",
            "\n",
            "Iteration 4200:\tlearning_rate=0.000467,\tloss_train=0.001796,\tloss_val=0.154208,\tacc_train=0.990000,\tacc_val=0.973810\n",
            "\n",
            "\n",
            "Iteration 4300:\tlearning_rate=0.000449,\tloss_train=0.001267,\tloss_val=0.188042,\tacc_train=1.000000,\tacc_val=0.966667\n",
            "\n",
            "\n",
            "Iteration 4400:\tlearning_rate=0.000432,\tloss_train=0.000136,\tloss_val=0.188587,\tacc_train=1.000000,\tacc_val=0.969048\n",
            "\n",
            "\n",
            "Iteration 4500:\tlearning_rate=0.000416,\tloss_train=0.027041,\tloss_val=0.192260,\tacc_train=0.990000,\tacc_val=0.971429\n",
            "\n",
            "\n",
            "Iteration 4600:\tlearning_rate=0.000401,\tloss_train=0.010266,\tloss_val=0.187360,\tacc_train=1.000000,\tacc_val=0.976190\n",
            "\n",
            "\n",
            "Iteration 4700:\tlearning_rate=0.000386,\tloss_train=0.019841,\tloss_val=0.191396,\tacc_train=0.990000,\tacc_val=0.973810\n",
            "\n",
            "\n",
            "Iteration 4800:\tlearning_rate=0.000372,\tloss_train=0.008633,\tloss_val=0.184235,\tacc_train=1.000000,\tacc_val=0.973810\n",
            "\n",
            "\n",
            "Iteration 4900:\tlearning_rate=0.000359,\tloss_train=0.014715,\tloss_val=0.178505,\tacc_train=1.000000,\tacc_val=0.973810\n",
            "\n",
            "\n",
            "Iteration 5000:\tlearning_rate=0.000346,\tloss_train=0.000880,\tloss_val=0.193680,\tacc_train=1.000000,\tacc_val=0.973810\n",
            "\n",
            "\n",
            "Iteration 5100:\tlearning_rate=0.000334,\tloss_train=0.000306,\tloss_val=0.196898,\tacc_train=1.000000,\tacc_val=0.971429\n",
            "\n",
            "\n",
            "Iteration 5200:\tlearning_rate=0.000323,\tloss_train=0.003086,\tloss_val=0.169877,\tacc_train=1.000000,\tacc_val=0.978571\n",
            "\n",
            "\n",
            "Iteration 5300:\tlearning_rate=0.000312,\tloss_train=0.001432,\tloss_val=0.179778,\tacc_train=1.000000,\tacc_val=0.978571\n",
            "\n",
            "\n",
            "Iteration 5400:\tlearning_rate=0.000302,\tloss_train=0.000128,\tloss_val=0.211507,\tacc_train=1.000000,\tacc_val=0.969048\n",
            "\n",
            "\n",
            "Iteration 5500:\tlearning_rate=0.000292,\tloss_train=0.000819,\tloss_val=0.204917,\tacc_train=1.000000,\tacc_val=0.976190\n",
            "\n",
            "\n",
            "Iteration 5600:\tlearning_rate=0.000282,\tloss_train=0.000926,\tloss_val=0.198197,\tacc_train=1.000000,\tacc_val=0.973810\n",
            "\n",
            "\n",
            "Iteration 5700:\tlearning_rate=0.000274,\tloss_train=0.002285,\tloss_val=0.184203,\tacc_train=1.000000,\tacc_val=0.976190\n",
            "\n",
            "\n",
            "Iteration 5800:\tlearning_rate=0.000265,\tloss_train=0.003996,\tloss_val=0.200789,\tacc_train=1.000000,\tacc_val=0.978571\n",
            "\n",
            "\n",
            "Iteration 5900:\tlearning_rate=0.000257,\tloss_train=0.000190,\tloss_val=0.197559,\tacc_train=1.000000,\tacc_val=0.973810\n",
            "\n",
            "\n",
            "Iteration 6000:\tlearning_rate=0.000249,\tloss_train=0.004597,\tloss_val=0.201104,\tacc_train=1.000000,\tacc_val=0.976190\n",
            "\n",
            "\n",
            "Iteration 6100:\tlearning_rate=0.000242,\tloss_train=0.000215,\tloss_val=0.202130,\tacc_train=1.000000,\tacc_val=0.976190\n",
            "\n",
            "\n",
            "Iteration 6200:\tlearning_rate=0.000235,\tloss_train=0.000377,\tloss_val=0.193507,\tacc_train=1.000000,\tacc_val=0.976190\n",
            "\n",
            "\n",
            "Iteration 6300:\tlearning_rate=0.000229,\tloss_train=0.001215,\tloss_val=0.200370,\tacc_train=1.000000,\tacc_val=0.976190\n",
            "\n",
            "\n",
            "Iteration 6400:\tlearning_rate=0.000222,\tloss_train=0.000771,\tloss_val=0.201266,\tacc_train=1.000000,\tacc_val=0.976190\n",
            "\n",
            "\n",
            "Iteration 6500:\tlearning_rate=0.000216,\tloss_train=0.000272,\tloss_val=0.205093,\tacc_train=1.000000,\tacc_val=0.976190\n",
            "\n",
            "\n",
            "Iteration 6600:\tlearning_rate=0.000211,\tloss_train=0.002038,\tloss_val=0.211768,\tacc_train=1.000000,\tacc_val=0.976190\n",
            "\n",
            "\n",
            "Iteration 6700:\tlearning_rate=0.000205,\tloss_train=0.000121,\tloss_val=0.206673,\tacc_train=1.000000,\tacc_val=0.978571\n",
            "\n",
            "\n",
            "Iteration 6800:\tlearning_rate=0.000200,\tloss_train=0.000104,\tloss_val=0.209474,\tacc_train=1.000000,\tacc_val=0.978571\n",
            "\n",
            "\n",
            "Iteration 6900:\tlearning_rate=0.000195,\tloss_train=0.000312,\tloss_val=0.213413,\tacc_train=1.000000,\tacc_val=0.976190\n",
            "\n",
            "\n",
            "Iteration 7000:\tlearning_rate=0.000191,\tloss_train=0.001025,\tloss_val=0.212398,\tacc_train=1.000000,\tacc_val=0.978571\n",
            "\n",
            "\n",
            "Iteration 7100:\tlearning_rate=0.000186,\tloss_train=0.000644,\tloss_val=0.222158,\tacc_train=1.000000,\tacc_val=0.976190\n",
            "\n",
            "\n",
            "Iteration 7200:\tlearning_rate=0.000182,\tloss_train=0.000678,\tloss_val=0.222249,\tacc_train=1.000000,\tacc_val=0.976190\n",
            "\n",
            "\n",
            "Iteration 7300:\tlearning_rate=0.000178,\tloss_train=0.000055,\tloss_val=0.227992,\tacc_train=1.000000,\tacc_val=0.976190\n",
            "\n",
            "\n",
            "Iteration 7400:\tlearning_rate=0.000174,\tloss_train=0.001005,\tloss_val=0.213649,\tacc_train=1.000000,\tacc_val=0.976190\n",
            "\n",
            "\n",
            "Iteration 7500:\tlearning_rate=0.000171,\tloss_train=0.000277,\tloss_val=0.223501,\tacc_train=1.000000,\tacc_val=0.976190\n",
            "\n",
            "\n",
            "Iteration 7600:\tlearning_rate=0.000167,\tloss_train=0.000132,\tloss_val=0.219531,\tacc_train=1.000000,\tacc_val=0.976190\n",
            "\n",
            "\n",
            "Iteration 7700:\tlearning_rate=0.000164,\tloss_train=0.002109,\tloss_val=0.217459,\tacc_train=1.000000,\tacc_val=0.978571\n",
            "\n",
            "\n",
            "Iteration 7800:\tlearning_rate=0.000161,\tloss_train=0.000165,\tloss_val=0.217044,\tacc_train=1.000000,\tacc_val=0.976190\n",
            "\n",
            "\n",
            "Iteration 7900:\tlearning_rate=0.000158,\tloss_train=0.000023,\tloss_val=0.229679,\tacc_train=1.000000,\tacc_val=0.973810\n",
            "\n",
            "\n",
            "Iteration 8000:\tlearning_rate=0.000155,\tloss_train=0.024321,\tloss_val=0.224189,\tacc_train=1.000000,\tacc_val=0.973810\n",
            "\n",
            "\n",
            "Iteration 8100:\tlearning_rate=0.000152,\tloss_train=0.000761,\tloss_val=0.235591,\tacc_train=1.000000,\tacc_val=0.973810\n",
            "\n",
            "\n",
            "Iteration 8200:\tlearning_rate=0.000150,\tloss_train=0.000607,\tloss_val=0.225933,\tacc_train=1.000000,\tacc_val=0.973810\n",
            "\n",
            "\n",
            "Iteration 8300:\tlearning_rate=0.000147,\tloss_train=0.000154,\tloss_val=0.237070,\tacc_train=1.000000,\tacc_val=0.971429\n",
            "\n",
            "\n",
            "Iteration 8400:\tlearning_rate=0.000145,\tloss_train=0.000206,\tloss_val=0.244491,\tacc_train=0.990000,\tacc_val=0.971429\n",
            "\n",
            "\n",
            "Iteration 8500:\tlearning_rate=0.000143,\tloss_train=0.000283,\tloss_val=0.242510,\tacc_train=1.000000,\tacc_val=0.973810\n",
            "\n",
            "\n",
            "Iteration 8600:\tlearning_rate=0.000141,\tloss_train=0.000283,\tloss_val=0.242604,\tacc_train=1.000000,\tacc_val=0.976190\n",
            "\n",
            "\n",
            "Iteration 8700:\tlearning_rate=0.000139,\tloss_train=0.000079,\tloss_val=0.245553,\tacc_train=1.000000,\tacc_val=0.978571\n",
            "\n",
            "\n",
            "Iteration 8800:\tlearning_rate=0.000137,\tloss_train=0.003777,\tloss_val=0.250749,\tacc_train=1.000000,\tacc_val=0.973810\n",
            "\n",
            "\n",
            "Iteration 8900:\tlearning_rate=0.000135,\tloss_train=0.000855,\tloss_val=0.262589,\tacc_train=1.000000,\tacc_val=0.971429\n",
            "\n",
            "\n",
            "Iteration 9000:\tlearning_rate=0.000133,\tloss_train=0.002863,\tloss_val=0.235896,\tacc_train=1.000000,\tacc_val=0.976190\n",
            "\n",
            "\n",
            "Iteration 9100:\tlearning_rate=0.000132,\tloss_train=0.000093,\tloss_val=0.235709,\tacc_train=1.000000,\tacc_val=0.978571\n",
            "\n",
            "\n",
            "Iteration 9200:\tlearning_rate=0.000130,\tloss_train=0.000384,\tloss_val=0.246252,\tacc_train=1.000000,\tacc_val=0.976190\n",
            "\n",
            "\n",
            "Iteration 9300:\tlearning_rate=0.000129,\tloss_train=0.000413,\tloss_val=0.257204,\tacc_train=1.000000,\tacc_val=0.976190\n",
            "\n",
            "\n",
            "Iteration 9400:\tlearning_rate=0.000127,\tloss_train=0.001143,\tloss_val=0.239449,\tacc_train=1.000000,\tacc_val=0.978571\n",
            "\n",
            "\n",
            "Iteration 9500:\tlearning_rate=0.000126,\tloss_train=0.001286,\tloss_val=0.247709,\tacc_train=1.000000,\tacc_val=0.976190\n",
            "\n",
            "\n",
            "Iteration 9600:\tlearning_rate=0.000125,\tloss_train=0.000069,\tloss_val=0.258161,\tacc_train=1.000000,\tacc_val=0.973810\n",
            "\n",
            "\n",
            "Iteration 9700:\tlearning_rate=0.000123,\tloss_train=0.000114,\tloss_val=0.240440,\tacc_train=1.000000,\tacc_val=0.973810\n",
            "\n",
            "\n",
            "Iteration 9800:\tlearning_rate=0.000122,\tloss_train=0.000167,\tloss_val=0.248497,\tacc_train=1.000000,\tacc_val=0.976190\n",
            "\n",
            "\n",
            "Iteration 9900:\tlearning_rate=0.000121,\tloss_train=0.000088,\tloss_val=0.245352,\tacc_train=1.000000,\tacc_val=0.976190\n",
            "\n",
            "\n",
            "Iteration 10000:\tlearning_rate=0.000120,\tloss_train=0.001925,\tloss_val=0.251792,\tacc_train=1.000000,\tacc_val=0.973810\n",
            "\n",
            "\n",
            "Iteration 10100:\tlearning_rate=0.000119,\tloss_train=0.000010,\tloss_val=0.244316,\tacc_train=1.000000,\tacc_val=0.976190\n",
            "\n",
            "\n",
            "Iteration 10200:\tlearning_rate=0.000118,\tloss_train=0.000363,\tloss_val=0.234244,\tacc_train=1.000000,\tacc_val=0.976190\n",
            "\n",
            "\n",
            "Iteration 10300:\tlearning_rate=0.000117,\tloss_train=0.004417,\tloss_val=0.235212,\tacc_train=1.000000,\tacc_val=0.976190\n",
            "\n",
            "\n",
            "Iteration 10400:\tlearning_rate=0.000117,\tloss_train=0.000004,\tloss_val=0.233202,\tacc_train=1.000000,\tacc_val=0.976190\n",
            "\n",
            "\n",
            "Iteration 10500:\tlearning_rate=0.000116,\tloss_train=0.000044,\tloss_val=0.242240,\tacc_train=1.000000,\tacc_val=0.976190\n",
            "\n",
            "\n",
            "Iteration 10600:\tlearning_rate=0.000115,\tloss_train=0.018054,\tloss_val=0.242421,\tacc_train=1.000000,\tacc_val=0.976190\n",
            "\n",
            "\n",
            "Iteration 10700:\tlearning_rate=0.000114,\tloss_train=0.000098,\tloss_val=0.246199,\tacc_train=1.000000,\tacc_val=0.978571\n",
            "\n",
            "\n",
            "Iteration 10800:\tlearning_rate=0.000114,\tloss_train=0.000015,\tloss_val=0.252720,\tacc_train=1.000000,\tacc_val=0.976190\n",
            "\n",
            "\n",
            "Iteration 10900:\tlearning_rate=0.000113,\tloss_train=0.000167,\tloss_val=0.248268,\tacc_train=1.000000,\tacc_val=0.976190\n",
            "\n",
            "\n",
            "Iteration 11000:\tlearning_rate=0.000112,\tloss_train=0.000853,\tloss_val=0.258680,\tacc_train=1.000000,\tacc_val=0.976190\n",
            "\n",
            "\n",
            "Iteration 11100:\tlearning_rate=0.000112,\tloss_train=0.001663,\tloss_val=0.260736,\tacc_train=1.000000,\tacc_val=0.978571\n",
            "\n",
            "\n",
            "Iteration 11200:\tlearning_rate=0.000111,\tloss_train=0.000010,\tloss_val=0.269438,\tacc_train=1.000000,\tacc_val=0.976190\n",
            "\n",
            "\n",
            "Iteration 11300:\tlearning_rate=0.000111,\tloss_train=0.000010,\tloss_val=0.271105,\tacc_train=1.000000,\tacc_val=0.969048\n",
            "\n",
            "\n",
            "Iteration 11400:\tlearning_rate=0.000110,\tloss_train=0.000041,\tloss_val=0.279866,\tacc_train=0.990000,\tacc_val=0.969048\n",
            "\n",
            "\n",
            "Iteration 11500:\tlearning_rate=0.000110,\tloss_train=0.000061,\tloss_val=0.275283,\tacc_train=1.000000,\tacc_val=0.971429\n",
            "\n",
            "\n",
            "Iteration 11600:\tlearning_rate=0.000109,\tloss_train=0.000049,\tloss_val=0.265061,\tacc_train=1.000000,\tacc_val=0.969048\n",
            "\n",
            "\n",
            "Iteration 11700:\tlearning_rate=0.000109,\tloss_train=0.000055,\tloss_val=0.271662,\tacc_train=1.000000,\tacc_val=0.971429\n",
            "\n",
            "\n",
            "Iteration 11800:\tlearning_rate=0.000108,\tloss_train=0.000045,\tloss_val=0.269833,\tacc_train=1.000000,\tacc_val=0.971429\n",
            "\n",
            "\n",
            "Iteration 11900:\tlearning_rate=0.000108,\tloss_train=0.000176,\tloss_val=0.274895,\tacc_train=1.000000,\tacc_val=0.976190\n",
            "\n",
            "\n",
            "Iteration 12000:\tlearning_rate=0.000107,\tloss_train=0.000036,\tloss_val=0.267733,\tacc_train=1.000000,\tacc_val=0.976190\n",
            "\n",
            "\n",
            "Iteration 12100:\tlearning_rate=0.000107,\tloss_train=0.000205,\tloss_val=0.282164,\tacc_train=1.000000,\tacc_val=0.976190\n",
            "\n",
            "\n",
            "Iteration 12200:\tlearning_rate=0.000107,\tloss_train=0.003514,\tloss_val=0.273464,\tacc_train=1.000000,\tacc_val=0.976190\n",
            "\n",
            "\n",
            "Iteration 12300:\tlearning_rate=0.000106,\tloss_train=0.000627,\tloss_val=0.282306,\tacc_train=1.000000,\tacc_val=0.976190\n",
            "\n",
            "\n",
            "Iteration 12400:\tlearning_rate=0.000106,\tloss_train=0.000305,\tloss_val=0.261437,\tacc_train=1.000000,\tacc_val=0.976190\n",
            "\n",
            "\n",
            "Iteration 12500:\tlearning_rate=0.000106,\tloss_train=0.000208,\tloss_val=0.283386,\tacc_train=1.000000,\tacc_val=0.976190\n",
            "\n",
            "\n",
            "Iteration 12600:\tlearning_rate=0.000106,\tloss_train=0.000146,\tloss_val=0.276381,\tacc_train=1.000000,\tacc_val=0.973810\n",
            "\n",
            "\n",
            "Iteration 12700:\tlearning_rate=0.000105,\tloss_train=0.000332,\tloss_val=0.275132,\tacc_train=1.000000,\tacc_val=0.978571\n",
            "\n",
            "\n",
            "Iteration 12800:\tlearning_rate=0.000105,\tloss_train=0.000091,\tloss_val=0.276774,\tacc_train=1.000000,\tacc_val=0.978571\n",
            "\n",
            "\n",
            "Iteration 12900:\tlearning_rate=0.000105,\tloss_train=0.000014,\tloss_val=0.273545,\tacc_train=1.000000,\tacc_val=0.976190\n",
            "\n",
            "\n",
            "Iteration 13000:\tlearning_rate=0.000105,\tloss_train=0.000001,\tloss_val=0.281091,\tacc_train=1.000000,\tacc_val=0.976190\n",
            "\n",
            "\n",
            "Iteration 13100:\tlearning_rate=0.000104,\tloss_train=0.000047,\tloss_val=0.307178,\tacc_train=1.000000,\tacc_val=0.973810\n",
            "\n",
            "\n",
            "Iteration 13200:\tlearning_rate=0.000104,\tloss_train=0.000950,\tloss_val=0.278284,\tacc_train=1.000000,\tacc_val=0.971429\n",
            "\n",
            "\n",
            "Iteration 13300:\tlearning_rate=0.000104,\tloss_train=0.001268,\tloss_val=0.270979,\tacc_train=1.000000,\tacc_val=0.973810\n",
            "\n",
            "\n",
            "Iteration 13400:\tlearning_rate=0.000104,\tloss_train=0.002112,\tloss_val=0.275955,\tacc_train=1.000000,\tacc_val=0.973810\n",
            "\n",
            "\n",
            "Iteration 13500:\tlearning_rate=0.000104,\tloss_train=0.005143,\tloss_val=0.267192,\tacc_train=1.000000,\tacc_val=0.973810\n",
            "\n",
            "\n",
            "Iteration 13600:\tlearning_rate=0.000103,\tloss_train=0.000056,\tloss_val=0.271092,\tacc_train=1.000000,\tacc_val=0.973810\n",
            "\n",
            "\n",
            "Iteration 13700:\tlearning_rate=0.000103,\tloss_train=0.000020,\tloss_val=0.283081,\tacc_train=1.000000,\tacc_val=0.973810\n",
            "\n",
            "\n",
            "Iteration 13800:\tlearning_rate=0.000103,\tloss_train=0.000229,\tloss_val=0.280981,\tacc_train=1.000000,\tacc_val=0.973810\n",
            "\n",
            "\n",
            "Iteration 13900:\tlearning_rate=0.000103,\tloss_train=0.000050,\tloss_val=0.278143,\tacc_train=1.000000,\tacc_val=0.973810\n",
            "\n",
            "\n",
            "Iteration 14000:\tlearning_rate=0.000103,\tloss_train=0.000001,\tloss_val=0.270836,\tacc_train=1.000000,\tacc_val=0.973810\n",
            "\n",
            "\n",
            "Iteration 14100:\tlearning_rate=0.000103,\tloss_train=0.010002,\tloss_val=0.269828,\tacc_train=1.000000,\tacc_val=0.973810\n",
            "\n",
            "\n",
            "Iteration 14200:\tlearning_rate=0.000102,\tloss_train=0.000095,\tloss_val=0.272578,\tacc_train=1.000000,\tacc_val=0.976190\n",
            "\n",
            "\n",
            "Iteration 14300:\tlearning_rate=0.000102,\tloss_train=0.000472,\tloss_val=0.294209,\tacc_train=1.000000,\tacc_val=0.976190\n",
            "\n",
            "\n",
            "Iteration 14400:\tlearning_rate=0.000102,\tloss_train=0.000030,\tloss_val=0.295307,\tacc_train=1.000000,\tacc_val=0.976190\n",
            "\n",
            "\n",
            "Iteration 14500:\tlearning_rate=0.000102,\tloss_train=0.000101,\tloss_val=0.297480,\tacc_train=1.000000,\tacc_val=0.973810\n",
            "\n",
            "\n",
            "Iteration 14600:\tlearning_rate=0.000102,\tloss_train=0.000110,\tloss_val=0.293045,\tacc_train=1.000000,\tacc_val=0.976190\n",
            "\n",
            "\n",
            "Iteration 14700:\tlearning_rate=0.000102,\tloss_train=0.000073,\tloss_val=0.307468,\tacc_train=1.000000,\tacc_val=0.976190\n",
            "\n",
            "\n",
            "Iteration 14800:\tlearning_rate=0.000102,\tloss_train=0.000063,\tloss_val=0.299858,\tacc_train=1.000000,\tacc_val=0.973810\n",
            "\n",
            "\n",
            "Iteration 14900:\tlearning_rate=0.000102,\tloss_train=0.000895,\tloss_val=0.299773,\tacc_train=1.000000,\tacc_val=0.976190\n",
            "\n",
            "\n",
            "Iteration 15000:\tlearning_rate=0.000102,\tloss_train=0.000041,\tloss_val=0.285019,\tacc_train=1.000000,\tacc_val=0.976190\n",
            "\n",
            "\n",
            "Iteration 15100:\tlearning_rate=0.000102,\tloss_train=0.000144,\tloss_val=0.280034,\tacc_train=1.000000,\tacc_val=0.976190\n",
            "\n",
            "\n",
            "Iteration 15200:\tlearning_rate=0.000102,\tloss_train=0.000375,\tloss_val=0.283464,\tacc_train=1.000000,\tacc_val=0.978571\n",
            "\n",
            "\n",
            "Iteration 15300:\tlearning_rate=0.000101,\tloss_train=0.000986,\tloss_val=0.267942,\tacc_train=1.000000,\tacc_val=0.978571\n",
            "\n",
            "\n",
            "Iteration 15400:\tlearning_rate=0.000101,\tloss_train=0.000005,\tloss_val=0.294002,\tacc_train=1.000000,\tacc_val=0.976190\n",
            "\n",
            "\n",
            "Iteration 15500:\tlearning_rate=0.000101,\tloss_train=0.000696,\tloss_val=0.285767,\tacc_train=1.000000,\tacc_val=0.978571\n",
            "\n",
            "\n",
            "Iteration 15600:\tlearning_rate=0.000101,\tloss_train=0.000081,\tloss_val=0.296837,\tacc_train=1.000000,\tacc_val=0.978571\n",
            "\n",
            "\n",
            "Iteration 15700:\tlearning_rate=0.000101,\tloss_train=0.000021,\tloss_val=0.293206,\tacc_train=1.000000,\tacc_val=0.978571\n",
            "\n",
            "\n",
            "Iteration 15800:\tlearning_rate=0.000101,\tloss_train=0.000022,\tloss_val=0.302308,\tacc_train=1.000000,\tacc_val=0.976190\n",
            "\n",
            "\n",
            "Iteration 15900:\tlearning_rate=0.000101,\tloss_train=0.003176,\tloss_val=0.301050,\tacc_train=1.000000,\tacc_val=0.971429\n",
            "\n",
            "\n",
            "Iteration 16000:\tlearning_rate=0.000101,\tloss_train=0.000607,\tloss_val=0.301576,\tacc_train=1.000000,\tacc_val=0.973810\n",
            "\n",
            "\n",
            "Iteration 16100:\tlearning_rate=0.000101,\tloss_train=0.000084,\tloss_val=0.302434,\tacc_train=1.000000,\tacc_val=0.973810\n",
            "\n",
            "\n",
            "Iteration 16200:\tlearning_rate=0.000101,\tloss_train=0.000108,\tloss_val=0.305228,\tacc_train=1.000000,\tacc_val=0.973810\n",
            "\n",
            "\n",
            "Iteration 16300:\tlearning_rate=0.000101,\tloss_train=0.000088,\tloss_val=0.304499,\tacc_train=1.000000,\tacc_val=0.973810\n",
            "\n",
            "\n",
            "Iteration 16400:\tlearning_rate=0.000101,\tloss_train=0.000067,\tloss_val=0.293064,\tacc_train=1.000000,\tacc_val=0.978571\n",
            "\n",
            "\n",
            "Iteration 16500:\tlearning_rate=0.000101,\tloss_train=0.000210,\tloss_val=0.294575,\tacc_train=0.990000,\tacc_val=0.978571\n",
            "\n",
            "\n",
            "Iteration 16600:\tlearning_rate=0.000101,\tloss_train=0.000028,\tloss_val=0.296519,\tacc_train=1.000000,\tacc_val=0.978571\n",
            "\n",
            "\n",
            "Iteration 16700:\tlearning_rate=0.000101,\tloss_train=0.000018,\tloss_val=0.297182,\tacc_train=1.000000,\tacc_val=0.976190\n",
            "\n",
            "\n",
            "Iteration 16800:\tlearning_rate=0.000101,\tloss_train=0.000097,\tloss_val=0.298227,\tacc_train=1.000000,\tacc_val=0.976190\n",
            "\n",
            "\n",
            "Iteration 16900:\tlearning_rate=0.000101,\tloss_train=0.000021,\tloss_val=0.297380,\tacc_train=1.000000,\tacc_val=0.976190\n",
            "\n",
            "\n",
            "Iteration 17000:\tlearning_rate=0.000101,\tloss_train=0.000016,\tloss_val=0.299017,\tacc_train=1.000000,\tacc_val=0.976190\n",
            "\n",
            "\n",
            "Iteration 17100:\tlearning_rate=0.000101,\tloss_train=0.008901,\tloss_val=0.304503,\tacc_train=1.000000,\tacc_val=0.976190\n",
            "\n",
            "\n",
            "Iteration 17200:\tlearning_rate=0.000101,\tloss_train=0.000045,\tloss_val=0.316930,\tacc_train=1.000000,\tacc_val=0.978571\n",
            "\n",
            "\n",
            "Iteration 17300:\tlearning_rate=0.000101,\tloss_train=0.000095,\tloss_val=0.311891,\tacc_train=1.000000,\tacc_val=0.978571\n",
            "\n",
            "\n",
            "Iteration 17400:\tlearning_rate=0.000100,\tloss_train=0.003854,\tloss_val=0.293427,\tacc_train=1.000000,\tacc_val=0.978571\n",
            "\n",
            "\n",
            "Iteration 17500:\tlearning_rate=0.000100,\tloss_train=0.000000,\tloss_val=0.307243,\tacc_train=1.000000,\tacc_val=0.976190\n",
            "\n",
            "\n",
            "Iteration 17600:\tlearning_rate=0.000100,\tloss_train=0.000018,\tloss_val=0.310439,\tacc_train=1.000000,\tacc_val=0.976190\n",
            "\n",
            "\n",
            "Iteration 17700:\tlearning_rate=0.000100,\tloss_train=0.000041,\tloss_val=0.316115,\tacc_train=1.000000,\tacc_val=0.976190\n",
            "\n",
            "\n",
            "Iteration 17800:\tlearning_rate=0.000100,\tloss_train=0.000066,\tloss_val=0.314930,\tacc_train=1.000000,\tacc_val=0.973810\n",
            "\n",
            "\n",
            "Iteration 17900:\tlearning_rate=0.000100,\tloss_train=0.000118,\tloss_val=0.332842,\tacc_train=1.000000,\tacc_val=0.973810\n",
            "\n",
            "\n",
            "Iteration 18000:\tlearning_rate=0.000100,\tloss_train=0.000021,\tloss_val=0.332029,\tacc_train=1.000000,\tacc_val=0.976190\n",
            "\n",
            "\n",
            "Iteration 18100:\tlearning_rate=0.000100,\tloss_train=0.003820,\tloss_val=0.321133,\tacc_train=1.000000,\tacc_val=0.976190\n",
            "\n",
            "\n",
            "Iteration 18200:\tlearning_rate=0.000100,\tloss_train=0.000195,\tloss_val=0.312012,\tacc_train=1.000000,\tacc_val=0.978571\n",
            "\n",
            "\n",
            "Iteration 18300:\tlearning_rate=0.000100,\tloss_train=0.000081,\tloss_val=0.331099,\tacc_train=1.000000,\tacc_val=0.976190\n",
            "\n",
            "\n",
            "Iteration 18400:\tlearning_rate=0.000100,\tloss_train=0.000006,\tloss_val=0.330951,\tacc_train=1.000000,\tacc_val=0.973810\n",
            "\n",
            "\n",
            "Iteration 18500:\tlearning_rate=0.000100,\tloss_train=0.000004,\tloss_val=0.333262,\tacc_train=1.000000,\tacc_val=0.976190\n",
            "\n",
            "\n",
            "Iteration 18600:\tlearning_rate=0.000100,\tloss_train=0.000274,\tloss_val=0.330258,\tacc_train=1.000000,\tacc_val=0.973810\n",
            "\n",
            "\n",
            "Iteration 18700:\tlearning_rate=0.000100,\tloss_train=0.000006,\tloss_val=0.331740,\tacc_train=1.000000,\tacc_val=0.973810\n",
            "\n",
            "\n",
            "Iteration 18800:\tlearning_rate=0.000100,\tloss_train=0.000005,\tloss_val=0.332040,\tacc_train=1.000000,\tacc_val=0.973810\n",
            "\n",
            "\n",
            "Iteration 18900:\tlearning_rate=0.000100,\tloss_train=0.000033,\tloss_val=0.329755,\tacc_train=1.000000,\tacc_val=0.973810\n",
            "\n",
            "\n",
            "Iteration 19000:\tlearning_rate=0.000100,\tloss_train=0.000215,\tloss_val=0.315635,\tacc_train=0.990000,\tacc_val=0.973810\n",
            "\n",
            "\n",
            "Iteration 19100:\tlearning_rate=0.000100,\tloss_train=0.000014,\tloss_val=0.313183,\tacc_train=1.000000,\tacc_val=0.973810\n",
            "\n",
            "\n",
            "Iteration 19200:\tlearning_rate=0.000100,\tloss_train=0.000023,\tloss_val=0.336975,\tacc_train=1.000000,\tacc_val=0.973810\n",
            "\n",
            "\n",
            "Iteration 19300:\tlearning_rate=0.000100,\tloss_train=0.000003,\tloss_val=0.334727,\tacc_train=1.000000,\tacc_val=0.978571\n",
            "\n",
            "\n",
            "Iteration 19400:\tlearning_rate=0.000100,\tloss_train=0.000004,\tloss_val=0.338605,\tacc_train=1.000000,\tacc_val=0.973810\n",
            "\n",
            "\n",
            "Iteration 19500:\tlearning_rate=0.000100,\tloss_train=0.011826,\tloss_val=0.342804,\tacc_train=1.000000,\tacc_val=0.973810\n",
            "\n",
            "\n",
            "Iteration 19600:\tlearning_rate=0.000100,\tloss_train=0.000003,\tloss_val=0.346977,\tacc_train=1.000000,\tacc_val=0.973810\n",
            "\n",
            "\n",
            "Iteration 19700:\tlearning_rate=0.000100,\tloss_train=0.000043,\tloss_val=0.339944,\tacc_train=1.000000,\tacc_val=0.973810\n",
            "\n",
            "\n",
            "Iteration 19800:\tlearning_rate=0.000100,\tloss_train=0.000009,\tloss_val=0.328747,\tacc_train=1.000000,\tacc_val=0.973810\n",
            "\n",
            "\n",
            "Iteration 19900:\tlearning_rate=0.000100,\tloss_train=0.000005,\tloss_val=0.329129,\tacc_train=1.000000,\tacc_val=0.973810\n",
            "\n",
            "\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "s5rC6lSIAR-P",
        "colab_type": "code",
        "outputId": "7e18fb26-18d7-4035-c327-c534664d0ecc",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 282
        }
      },
      "cell_type": "code",
      "source": [
        "plt.plot(history['acc_train'],'b')\n",
        "plt.plot(history['acc_val'],'r')"
      ],
      "execution_count": 72,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[<matplotlib.lines.Line2D at 0x7f651e90ecc0>]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 72
        },
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXIAAAD4CAYAAADxeG0DAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJzt3XmYHGW5/vFvL7NkwoQMZCALuyQv\nYZVEMGFLIKgQZAlEQdFDEPWowR+4IEFE5chhFSOLC8gmXHJQhISAbAfckAiGAEEQHgghEUg4DMkk\nmWQyS3fV74+q7unp6ZnpnkzPTMH9ua650l1VXX1Ppebpt996qyrm+z4iIhJd8cEOICIiW0eFXEQk\n4lTIRUQiToVcRCTiVMhFRCIuOdBv2NDQ1OdhMnV1NTQ2NvdnnH4zVLMpV2mGai4YutmUqzR9zVVf\nXxvrbl6kWuTJZGKwI3RrqGZTrtIM1VwwdLMpV2nKkStShVxERLpSIRcRiTgVchGRiFMhFxGJOBVy\nEZGIK2r4oXNuX+A+YL6ZXZ8372jgUiANPGhmP+r3lCIi0q1eW+TOueHAdcDj3SxyLXAKcCjwcefc\n3v0XT0REelNMi7wVmAmcnz/DObcHsM7M3gyfPwjMAP7VnyG3xjPPxHnhhQRf+EJ7r8vefHMFu+/u\ncdRR6aLW/fjjCe69twKA6mpoaakuOtdxx6WYOTPFrbdW8MwzvY8rnTgxzdy57fz1rwnuvruCWAzO\nPLONAw7w+OlPKzn00DQf/Wia+fMref31js/nUnMNlFJzjRzpc8EFrTQ0xLjmmkra27s9N2JAc/Vm\n7FiP889v46WX4tx8cyXp4natAcnWX8qRK7Pd/vWvODfd1LftVo5c1dU+55/fRiIBl19eSXNz8fth\nPA5z5rRxzDH9Ging+35RPxMmTPjhhAkTzs6bdsiECRMW5Dw/a8KECZf2tJ729pQ/kI46yvfB9996\nq+flFi0Klps4sbj1ep7v77pr8Jq+/Awb5vu//nVpr7nxRt/fbruO57vv7vtXXRU8Hj3a96+/vu95\novBz7rm+f9hhg5+j1J/LL/f9Pfcc/BxR+7nsMt8fP37wc+T/nHKK73/2s3177UUXbVU5o7uf/j5F\nv9ePp605Zba+vpaGhqaSXvPSS8OBOE891cwRRxT+WN+8GebODZZ75RWfN97YRDwefIJWVwfz16yJ\nUVkJO+/sE4vBiy/GWbVqODNntnPJJa1sv/02rF27qahMjz2W5DvfqeaMMyAW87nrri2MH+91u/zq\n1TFmzarhy18ONu+FFwYt0xtvrOS884Jl3nkHzj4bhg3zue++ZkaN8gFKyjWQSsmVSsGnP13DT38a\nfNPIbPPBztWbTZtinHBCDfPmBf9vZ57Zxte/3jYksvWn/s61eXOw3S64YOu2Wzm211e/Ws099wRl\n88AD09x00xZiRTbK43EYM8YHSq9jENS/7mxtIV8NjM55Pi6cNiQ0NcH//V/wx798ebzbQn777RW8\n+WackSN91q+P8fzzCc4/v4qaGliwoJmPf7yG5cuD7o/LL2/hC19o5+GHg0134okpdtrJp74+KKLF\nOOOMdu67L8mTTyaZM6edI4/s+XvjTjv5fP3rbfzkJ1Xst1+auXPbaG2FBx9M8tZbcS65pIVf/7qC\n115L8K1vtfHhD3d8KJSSayCVmuvyy1s47bQahg/3ufTSVsaOLc/v1L/by+f732/lm9+sZswYj4su\namWbbYZKtv7T/7mC7faNb2zddivH9rryylZmzEjgeXDVVS3svPPQ+P/YqkJuZiudcyOcc7sBbwGf\nBE7vj2D9YfnyeMHH+ZYtC4r02We3ccklVdmiCHDccUERP+SQFC+9lOCSS6qYOTPFww8nqajwmTEj\nVXKuWAyuv76FO+6o4Oyzi2tpnHtuG8kkzJrVTjIJySTccssWnngiyVlntXPEEWkeeijJV7/a9xbf\nUHbUUWmuvrqFceO8shXxcvjsZ9vZsAGmTElvVRH/oPnMZ9pZv37obbeJEz1uuKGFVAr237/7b9ED\nLeb7Pf9ROOcmA1cDuwHtwNvAIuANM1vgnDsCuCJc/B4z+3FP69uaqx8W27XyyCMJXn89zqhRPmef\nPQyA6dNT/O53Wwouf/TRNbz6apw//3kzU6Z07DXxuI/nxRgzxuPJJzdz770VfPvb1ey1V5pXXkkw\nbVqKu+/eUlK2gZbJFV+1kuQ/X8Cvq6P9kMMo9H0wvuJ1Ylu2kN5n3+LfwPdJPvsM8TVrYFg1bYcc\nTizVTvKFZbQfcCDd/RVmc/17FfGGd0kdODn47lmCWNNGks8/R/vUQ4NPtoyWFiqefYbUxL3x67YL\nfrci36fY/8fE8teIbWkmte/+nbbl1vw+vamvr6Xh3Y1U/P1JYuvW4Y8aRfvBU/r9ffqaq8t+kE5R\nsfhv0BYONKiooH3qIfgjtu1xfbH1jVQsfhK8oFD6tbXBPltRUXj5TU0F32fUh3bq/f8ynSb57DN4\nY8bi7bRzSb93lueRXPIPUvsfAMOG9bp4X2tFT1c/7LVFbmZLgek9zP8rMLXkVGXi+zBvXjVvvx3n\n5JM7Rqp01yL3/WDeHnt47L67z3bbeaxbF6e62ueKK1r4wbdSXHllgm22gc99rp17702yeHGw2U49\ntetImFjjOmKNjeGTWLBz5O+AmzdDZWUwvbmZ+DtroKoKb+y4LgU2/s4aaA6PK1RWdtnZYu++y/D/\n/iHe6NG0Hj8Lv6am83vV7kXyuaWMPGkmsS3Bh87med+j5aRTGH75j2g/eAr+NrVU33kHlU8txo/F\naD73W8TSHsnnn+txWwPE31xF8o0V2efeiG2JpdqJNTfjDd+G1KTJECuw7SsTjGxcT8XSZwBI77wL\nLaedTutxJ+BXdx5p4O20c7C9AHyfiqf/TvWdd1C1aAGx5mbaDjmMphtuwdt+FDXzr2LYjb8gvmE9\nflUV7ZMPItbaUvB9iMWoeugBKp5aDJ4f/CEePZ3a//ldUJC6EVu3looXXwAgted4vLE7BdM3N3V6\nn02X/5i2j/U8RKHiib8w7JfXE2vt5ZtUIg7HHUvtE4upvu/e7OT0zruQ3v1DeHV1tM6aTdvHPgEV\nFZ33wyLE8En+42mqHryfWHPY4InHaDv0cLxdd6Pq/vuIbdiAN3IkrSecRHL5a9ntRmWCuhVvkFzx\nenZ9uftBLn/YMFpOO51NP7gEEgnib7/V+ddctZLac+eSWNO5h9ar34HUxH26BvfSVDz7TMH3YcoU\ntg17Lb0dd6Rl9qmkd92N5EsvUr3wHmIbNpB4/TUSb7+FH4vRPu1IWk47nfjbb5N8cRltR3+C1OSP\n4Pdw6C+WSjH8R9+n6pGHSE3cm6afXIcXNh66iMfxdt2t23VtjV5b5P2t3C3yF16Ic/TRwwFIJHzS\n6RhuzzYOXX4HFz9+IFX7jQcgnYb33osx4p47+PQPD2THEyZzx7G38/ClL/Lqm8N576PHcNkeN1D9\n2ztp+tmNtJ7yaSBoJGxcuY7t77qR6paNeKPH0HrybLbfdzyb/vsKhl98EbFUR3eLV78Drcd+Er+m\nhtT+B+BXVVH7zf+HN2oUW+aew/BLLyb+3nsApPbZj7bDDg8Kn+9TseQpKp5d2un323LGWWy68ifQ\n1kbVw39g+EUXkHin+6LDttvixePENmyg+RvnUf2b24mvW4s3egyJf6/qtGjb4dNJrFxB4s1/F/cf\nAvjV1bTOPJ7UpMnE16yhauE9UFFB27SjqHz8URJvvdnj69umHIK3y65UPbCIWPPmgst4221H68zj\n8YcNo/KPj5F8fTkA6V12I73rrlQ+8Rf8YcNI77IrSXslu80rnnqS5KtW9PsU/TvH47QdOQN/m1qq\nHv4DsdaOA6/Z91m0AID19z5A6oADqXz8f0m8arR9/Bjiq9+i8om/Elu3lurf3knMK+0revvBU2g9\ncRaJl16k6r4FxDd3HNDzRtWT2mdfKv72V2JbM9axRJ32g3feoWrB76GigpaTZ+NvPwqA2Lp1VP/+\ntyRWrSS92+7EGhuJb1jfdV3xOFv+cy7eTsEHZPyNFVTf8zvi3XwwpXfZjZbZn+r0PlX33t2pgdEd\nb5ta2j55AonXl1Ox5Om+/vqk9vhQpw+y7mw+99sMn39Vv7fI33eF/IorKrn66qrs8+HDfeYfeCtf\n+ttZtG63I5sffIj4qlV89+6DWLywkedT+7GJ4Syd9AWmPXtdwXV6I0ey/g+PkXzpn8Samqi55upO\nRdCPxYhVVUFLS1BEPvYJiMWItbRQ+cf/7bID+pWVxNqCFphfUUHrSacQ27Ceyj8+1ulDwI/HaT9i\nOulxwQ5dseRpkq8arTM+RsVzS4mvW4efSLD5uz/A23FHKp7+e/brKAStheon/gyrV9N06ZW0fPEr\nVD78INv+x2kANP/nXLxx44i1tNBy8qfwdtmV2Lq11Pz8OlJ770PrJ2Z2tIS7E49Doptx8L4fDDkp\noL6+lob3NmW7RGKbmqi6bwHJpUuC12V+h7Y2Kv/0OPH3GoJVVlfTetwJtJz+H9kuourbbqbmF9eR\nWPkGrcd+kqZrf46/7chgBe3twbecTNfLpk1UL1pA8pl/AJA6cDKtJ5yEn6yg6rFHGLH8ZRoPPSr8\nJtHN303u75xOd2zznPepfPxRRnzu1OB3SSSItRc+jyE9bic23ngrqQ9PKvxeme2wYQOjHnuATQ3r\n2fKVuR3f8jwP0mkS9grV/3MH1b//LfHGRtr3/zCp/fbvcZ35vHE70TL7VLxwf4ttaqJq0ULi76yh\n9YRZpPccT8JeoeqBhXjjdg62W83w4P9y7ebu94Ncra1s84PvMuyWX5HeYUfapx+Fn/uNNZ6g9VOn\n0j7lkLxwwe9ZUDLZ9f/K96mvGxbUC98nuey54Bvcpk3424+i5ZRPk95zfJA57JpKvGpULbwHb1Q9\n7Qd9lKoH7w++Efcivfc+bPnCl6l8+EEqH3uk+wXjcVo+P4e6j01TIe9xA/g+Zx76Fu+ubKFij3Es\nfnVHPrL/Fh59c2+2bVxFnI63/jPT+AcH8x2uyk7z6nfgrR/fyt8f3szxm+6CsWPwxoxlmx9e2Plt\nYjGav3EebcfMJPn8c1Tdv5DKlmZaxu7M5ksux9sxZyBPSwvJ1wxaWqj830dI/nMZm793MYlVK6n+\nza9p/tb5pCZ9BIDY2rUk3upoDadHj8Xfccfs8/j/vcPIY44i8fZbeKNG0fKpz9DyuTNIj5/Q/Tar\nG8baFwxv512y02quvgJaW2me971B618tqZ+wvZ3EKy8T89Kkd9+jcB+r7xNf+Qbebrt3X4D7O1cv\nqu69m2G/+iX4HqkDJ9N+wIFUPfwg3qh6Wk86Gb+2ltR4B/ndYVuTrbWV+HsN2WI8EPqyzeKrVgYZ\nk+W7SdlQP27Vh9e9Twu574PnEVu7ln+efze7/+l2dmt+GYC25DC+krqe4ye8zKxXf8y1fJ2qkdWc\ntOUuNqZqGJ82mhlGC9X8iIu4ZKdf0HbTDdmimuV5jJhzOolVb9A6azbpHUeT3mticDCrp2xlEl/9\nNolX/kX7YdN6by0PYK5SKVfphmo25SqNCnneBhjx+VOpeuSh7PNWKnmA49lrxg7s9dRvSGzeCEB6\n25Hsn3iJf60bC8ABPM/zHAjAz/kqc/k5K1Y0beUY3/fXTlNuylW6oZpNuUozKKNWhqr4v1dR9chD\npHcczdPN+3NX03Ec+atZTDl2JJWVsGHFl6i5/BJS++xH62mn8/ioWtrbg40Xi32IlrNPpvq+e7mV\nMxkzxhtSY1VFREoR2UJe9Yf7AfjHsRdy6G1nM2tWO4ed2JKdn95jT5puvC37PEHnYzFNP/0ZK2b9\nP56ZcxDTXekn9YiIDBXRLeQP3Icfj3Nf7CQAPv/53q9u2Mnw4dTPnMTttzezxx7ROVNQRCRfJAt5\n/J01VCx5mrZDDuPJ10YTi/kccEDfxswec8zAjbUVESmHSN7qrWrBPQC0zDyBZcsS7LmnR233FwYT\nEXlfi14hT6epvvlG0lXDeGXSqTQ1xTpd7U9E5IMmcl0rlY8+TPLfK7mRL/Gr748D4MMfVveIiHxw\nRa6QD7vpBgCu4+u8GN4iTYVcRD7IIte1knjheV5hL1bUBJdaTSZ99t1XXSsi8sEVuRZ5utVjC9V8\n7WttLF8ep6bGL+YSwCIi71vRK+QpD484xx6bYr/91BIXESmqkDvn5gNTAB84x8yW5Mw7Efge0Arc\nZWbXlyNohp/ySFTE1Z0iIhLqtY/cOTcNGG9mU4GzgGtz5sWB64GZwBHA8c65sl4/M4ZHLBnfmiuV\nioi8rxRzsHMGsBDAzF4G6pxzI8J5o4D1ZtZgZh7wOHB0WZKG4nj4hW4dJiLyAVVM18poIPd+Yw3h\ntI3h41rn3HhgJXAk8OeeVlZXV0MyWcSdRLqRIA3xBPX1Q+9UzqGYCZSrVEM1FwzdbMpVmv7O1ZeD\nndlODTPznXNnALcAG4A3cucX0tjY3NPsHtXX1xLHwyM25K4z/H679nG5KVfphmo25SrNVlyPvNt5\nxRTy1QQt8IyxQPZGdmb2F+BwAOfcZQQt8/LwfeL4+NEb/i4iUjbFVMRHgdkAzrlJwGozy36cOOce\ncs7t4JwbDhwPPFaWpJC9Ka8X73vXjIjI+02vLXIzW+ycW+qcWwx4wFzn3Bxgg5ktAH5FUOx94DIz\ne69sacO7letgp4hIh6L6yM1sXt6kZTnz7gXu7c9Q3fFT6aADXoVcRCQrUhUx3R60yD0VchGRrEhV\nRC+lrhURkXyRqoiZFrm6VkREOkSqImZa5Bq1IiLSIVKFPN0W3kBCLXIRkaxIVUQ/nekj1xWzREQy\nIlXIM33kfjxSsUVEyipSFVGjVkREuopURcwWch3sFBHJimQh18FOEZEOkaqIXnswakV95CIiHSJV\nETOjVtQiFxHpEKmKqFErIiJdRaoiZseRq5CLiGRFqiLqYKeISFeRqog62Cki0lVRN5Zwzs0HphDc\nBegcM1uSM28u8DkgDTxjZueWIyjoYKeISCG9VkTn3DRgvJlNBc4Crs2ZNwI4DzjczA4D9nbOTSlX\n2GzXilrkIiJZxVTEGcBCADN7GagLCzhAW/izjXMuCdQA68oRFHJa5CrkIiJZxXStjAaW5jxvCKdt\nNLMW59zFwApgC3CXmb3a08rq6mpIJvt2in1D2CJPVFVSX1/bp3WU01DMBMpVqqGaC4ZuNuUqTX/n\nKqqPPE/2GrJhy/y7wARgI/BH59wBZrasuxc3Njb34S1DqXT4j09DQ1Pf11MG9fW1Qy4TKFephmou\nGLrZlKs0fc3VU/Evpo9iNUELPGMssCZ8PBFYYWbvmVkb8AQwueSERVIfuYhIV8VUxEeB2QDOuUnA\najPLfJysBCY654aFzz8CvNbfITO8bB+5biwhIpLRa9eKmS12zi11zi0GPGCuc24OsMHMFjjnrgL+\n5JxLAYvN7IlyhfXVIhcR6aKoPnIzm5c3aVnOvBuAG/ozVHc6TtHX9chFRDIi1bTNnNmpFrmISIdI\nVUSNIxcR6SpSFTEzaiWWiFRsEZGyilZF9HTzZRGRfJGqiNlx5GqRi4hkRaoi+mkd7BQRyRetiphW\ni1xEJF+kKmL2YKda5CIiWZGqiL6n4YciIvkiVRGzp+gndGaniEhGtAq5WuQiIl1EqyKG1yPXCUEi\nIh0iVRGzp+gndBlbEZGMSBZyjVoREekQqYqoi2aJiHRV1PXInXPzgSmAD5xjZkvC6eOA3+Qsugcw\nz8zu7O+gkNMi7+PNm0VE3o96LeTOuWnAeDOb6pybCNwCTAUws7eB6eFySeDPwKJyhUWn6IuIdFFM\nRZwBLAQws5eBOufciALLzQHuMbNN/Revs44WuQq5iEhGMRVxNNCQ87whnJbvi8DN/RGqW5lx5LqM\nrYhIVlF95Hm6jP1zzk0FXjGzjb29uK6uhmQf+7gzLfLhI6qpr6/t0zrKaShmAuUq1VDNBUM3m3KV\npr9zFVPIV9O5BT4WWJO3zCeBx4p5w8bG5uKSFRK2yLe0pmhoaOr7esqgvr52yGUC5SrVUM0FQzeb\ncpWmr7l6Kv7F9FE8CswGcM5NAlabWX6Kg4BlJScrUeZ65DqzU0SkQ68V0cwWA0udc4uBa4G5zrk5\nzrlZOYuNAd4tU8YOad2zU0QkX1F95GY2L2/Ssrz5+/Vboh7oolkiIl1FqyKqRS4i0kW0KqKnceQi\nIvmiVRF1sFNEpItoVUT1kYuIdBGtihgW8nhS1yMXEcmIZCFX14qISIdIVcTMKfpxHewUEcmKVEWM\neeFlbBO6HrmISEakCrm6VkREuopWRVQhFxHpIloV0VMfuYhIvmhVRLXIRUS6iFZFzLTIK6IVW0Sk\nnCJVEbOjVnRmp4hIVrQqovrIRUS6iFZFVCEXEemiqBtLOOfmA1MAHzjHzJbkzNsZ+B+gEnjWzL5S\njqBAzsFOXWtFRCSj16atc24aMN7MpgJnEdzuLdfVwNVmdjCQds7t0v8xQ37mYKfO7BQRySimj2IG\nsBDAzF4G6pxzIwCcc3HgcGBROH+umf27TFmJe7oeuYhIvmK6VkYDS3OeN4TTNgL1QBMw3zk3CXjC\nzC7oaWV1dTUkk31sUYddK9vX17JdfW3f1lFG9UMwEyhXqYZqLhi62ZSrNP2dq6g+8jyxvMfjgGuA\nlcAfnHPHmdkfuntxY2NzH94yFHatrN/YTLqhqe/rKYP6+loahlgmUK5SDdVcMHSzKVdp+pqrp+Jf\nTB/FaoIWeMZYYE34+D1glZm9bmZp4HFgn5ITFikWtsgTOiFIRCSrmIr4KDAbIOw+WW1mTQBmlgJW\nOOfGh8tOBqwcQYFsizzW164ZEZH3oV67VsxssXNuqXNuMeABc51zc4ANZrYAOBe4LTzw+U/g/nKF\nzZzZqXHkIiIdiuojN7N5eZOW5cxbDhzWn6G6E/N10SwRkXyRqojqIxcR6SpaFdHXmZ0iIvkiVchj\nutaKiEgXkaqIcT88s1OjVkREsiJVyDNdK34sWrFFRMopUhUxM2pFN5YQEekQqYqY6SNXIRcR6RCp\niqgWuYhIV5GqiLHwYCcJHewUEcmIVCGPq0UuItJFpCpitmslphOCREQyIlfI09GKLCJSdpGqijHf\nw4tWZBGRsotUVQxa5DrQKSKSK1KFPE5aZ3WKiOSJVFVU14qISFdF3VjCOTcfmAL4wDlmtiRn3krg\nTSAc5M3pZvZ2/8YMqJCLiHTVayF3zk0DxpvZVOfcROAWYGreYsea2aZyBMwV9z11rYiI5CmmKs4A\nFgKY2ctAnXNuRFlTdSOGWuQiIvmK6VoZDSzNed4QTtuYM+2XzrndgL8BF5iZ393K6upqSPbxeuJr\n/TReLEF9fW2fXl9uylUa5SrdUM2mXKXp71xF9ZHnyT+t8vvAw8A6gpb7KcDvu3txY2NzH94yEA9b\n5A0NTX1eR7nU19cqVwmUq3RDNZtylaavuXoq/sUU8tUELfCMscCazBMzuz3z2Dn3ILAfPRTyrRFT\nH7mISBfFVMVHgdkAzrlJwGozawqfb+uce8Q5VxkuOw14sSxJCVvkKuQiIp302iI3s8XOuaXOucWA\nB8x1zs0BNpjZgrAV/pRzbgvwHGVqjYNa5CIihRTVR25m8/ImLcuZdw1wTX+G6k6C4GCnTtIXEekQ\nqeatTggSEekqUlUxjrpWRETyRaoqqpCLiHQVmaro+yrkIiKFRKYq+n7HwU4REekQmUKeToct8i4n\nloqIfLBFppB7nrpWREQKiUxVzLbIVchFRDqJTFVUi1xEpLDIVEXP08FOEZFCIlXI1SIXEekqMlUx\nnY6pkIuIFBCZqphtkccjE1lEZEBEpiqqa0VEpLDIVEUv7RPHVyEXEckTmaqYbvcA8DVqRUSkk6Ju\nLOGcmw9MAXzgHDNbUmCZy4CpZja9XxOGvHRYyNVHLiLSSa9V0Tk3DRhvZlOBs4BrCyyzN3BE/8fr\n4IUtctS1IiLSSTFVcQawEMDMXgbqnHMj8pa5Griwn7N14mdb5LpolohIrmK6VkYDS3OeN4TTNgKE\nN2L+C7CymDesq6shmSy9n3tdbfCZE08kqa+vLfn1A0G5SqNcpRuq2ZSrNP2dq6g+8jzZJrFzbjvg\nTOBoYFwxL25sbO7DW8K6hk0ApIjR0NDUp3WUU319rXKVQLlKN1SzKVdp+pqrp+JfTNfKaoIWeMZY\nYE34+CigHngCWABMCg+M9jsvpT5yEZFCiqmKjwKzAZxzk4DVZtYEYGa/N7O9zWwKMAt41sy+UY6g\nmUKuUSsiIp31WhXNbDGw1Dm3mGDEylzn3Bzn3Kyyp8vlqUUuIlJIUX3kZjYvb9KyAsusBKZvfaTC\nOk4IUiEXEckVmaroZ7pWEjqzU0QkV2QKecfBTo0jFxHJFZlC7qf94IEOdoqIdBKZqujrWisiIgVF\npipmCrlGrYiIdBaZqui1p4MHapGLiHQSmaqY7VrRqBURkU4iV8jVtSIi0llkqmK2kKtrRUSkk8hU\nxew4cl2PXESkk8gUctQiFxEpKDJV0dMp+iIiBUWmkGeufhhTi1xEpJPIVEWd2SkiUlhkqqJGrYiI\nFBaZqpi5jK26VkREOivqxhLhfTinAD5wjpktyZn3JeAsIE1ww4m5Zub3d1Cd2SkiUlivzVvn3DRg\nvJlNJSjY1+bMqwFOAw43s0OBvYCpZUmaPdipceQiIrmK6aeYASwEMLOXgTrn3IjwebOZzTCz9rCo\nbwu8U46guh65iEhhxXStjAaW5jxvCKdtzExwzs0DzgF+amYrelpZXV0NyWTp3SPDqoKoVTVV1NfX\nlvz6gaBcpVGu0g3VbMpVmv7OVVQfeZ4ufRtmdrlz7hrgQefc38zsye5e3NjY3Ie3hM1NLQC0tqdp\naGjq0zrKqb6+VrlKoFylG6rZlKs0fc3VU/Evpp9iNUELPGMssAbAObedc+4IADPbAjwEHFpywmKk\ng+uRx3SwU0Skk2IK+aPAbADn3CRgtZllPk4qgNucc9uEzw8GrN9TonHkIiLd6bVrxcwWO+eWOucW\nAx4w1zk3B9hgZgucc/8F/Mk5lyIYfrioLEnDUSskVMhFRHIV1UduZvPyJi3LmXcbcFv/RSos0yLX\nCUEiIp1FpipmC3lC48hFRHKBOTVYAAAHPUlEQVRFppDvPDYFwHajVMhFRHJFppDvt29QyEeNjkxk\nEZEBEZmqGPN082URkUKiUxU9DT8UESkkOlVRhVxEpKDoVMXwzE4VchGRzqJTFT1dj1xEpJDoFHI/\nvIxtTMMPRURyRaaQx9RHLiJSUHSqogq5iEhB0amKOtgpIlJQdKpi9uqHOtgpIpIrOoXcD0etqEUu\nItJJZKqiTtEXESmsqOuRO+fmA1MAHzjHzJbkzDsSuAxIE9wd6Itm5vV7Uh3sFBEpqNeq6JybBow3\ns6nAWcC1eYvcCMw2s0OBWuCYfk8JoFu9iYgUVExVnAEsBDCzl4E659yInPmTzeyt8HEDsH3/Rgxl\nW+Q6IUhEJFcxhXw0QYHOaAinAWBmGwGcc2OAjwMP9mfAjLYjZ8CJJ9I++eByrF5EJLKK6iPP06VJ\n7JzbAbgf+JqZre3pxXV1NSSTfRhCOH0qTF/IqNJfOWDq62sHO0JBylWaoZoLhm425SpNf+cqppCv\nJqcFDowF1mSehN0sDwEXmtmjva2ssbG51IxZ9fW1NDQ09fn15TRUsylXaYZqLhi62ZSrNH3N1VPx\nL6Zr5VFgNoBzbhKw2sxyU1wNzDezh0tOJiIiW63XFrmZLXbOLXXOLQY8YK5zbg6wAXgE+A9gvHPu\ni+FL7jSzG8sVWEREOiuqj9zM5uVNWpbzuKr/4oiISKk0KFtEJOJUyEVEIk6FXEQk4lTIRUQiLuZn\n7oUpIiKRpBa5iEjEqZCLiEScCrmISMSpkIuIRJwKuYhIxKmQi4hEnAq5iEjE9eXGEoOipxtAD1Ke\nK4HDCbbhZcAJwGQgc2ONq8zsDwOcaTpwN/BSOOmfwJXAHUCC4Drynzez1gHOdRbw+ZxJHwGeAYYD\nm8Np3zKzpQOYaV/gPoJLMF/vnNuZAtvJOXc6cC7BlT9vNLObByHXrUAF0A58zszecc61A0/mvHSG\nmaUHMNdtFNjfh8D2uhuoD2dvBzwFXErwt5DZvxrM7FNlzpVfH5ZQxv0rEoU89wbQzrmJwC3A1EHM\ncySwb5hne+A54I/ABWb2wGDlCv3FzGZnnjjnbgV+ZmZ3O+cuBb4A/GIgA4U7581hnmnAp4F9gDPN\n7MWBzBJmGA5cBzyeM/m/yNtOzrnbge8DBwNtwBLn3AIzWzeAuS4h+AP/nXNuLvBN4DvABjObXo4c\nReaCvP09XG5Qt1dugXbO3QLc1DFrwLZXofrwOGXcv6LStdLbDaAH2l+BzA6znqBl2Yf71w2I6cCi\n8PH9wNGDFwUIdtwfDXKGVmAmwd2vMqbTdTt9FFhiZhvMbAtBC/jQAc71NeCe8HH5bm7es0K5ChkK\n2wsA55wDRprZP8r4/t0pVB+mU8b9KxItcoJbzeV+7c7cAHrjYIQJv8JmugTOIrjhdBo42zn3TeBd\n4Gwze28Q4u3tnFtE8LXyYmB4TlfKu8CYQcgEgHPuIODNsGsA4L+cc6OAl4Fzw5257MwsBaTCDBmF\ntlP+jcfLuv0K5TKzzQDOuQQwl+CbA0C1c+5OYFfgHjP7yUDmCnXa3xkC2yvHOQSt9YzRzrnfE9yq\n8mdm9psy5ipUHz5Rzv0rKi3yfF1uAD0YnHMnEvxHnU3Q/zXPzI4Cngd+OAiRXiMo3icCZxB0Z+R+\nWA/2dvsicFv4+BrgPDM7gvDOU4MVqoDuttOgbL+wiN8B/NHMMt0I3wa+DHwcON0595EBjlXM/j5Y\n26sSOMzM/hROWgtcBHyG4FjWj5xzZW/Q5NWHXP2+f0WlRd7jDaAHg3PuE8CFwDFmtoHO/YeLGOB+\naAAzexv4bfj0defcO8BBzrlhYWt3HL1/PS6n6cDXAcxsQc70+4FTByNQjk0FtlP+fjeO4ODZQLsV\neM3MLs5MMLNfZh475x4H9iM4gDwgcj5QoGN//z1DY3tNA7JdKuE9hm8Nn77nnHsG2Isy1pD8+uCc\nK+v+FZUWeW83gB5QzrltgauAT2YOTDjn7nHO7REuMh0YjIN4pzvnvh0+Hg3sSLADnxIucgowKDfJ\nds6NBTaZWZtzLuace8w5NzKcPZ1B2F55HqPrdnqa4INwpHNuG4L+yycGMlQ4qqHNzH6QM8055+4M\nt2MyzPVStyspT65C+/ugb6/QQeTcjtI5d6Rz7ifh4+HAh4FXy/XmheoDZd6/InMZW+fc5UD2a7iZ\nLevlJeXM8mWCr5K5O8OtBF+hmoFNBCMy3h3gXLXAncBIoJKgm+U54HagGlgV5mofyFxhtsnAJWZ2\nbPj808D5BH2JbwNnmVnzAGa5GtiNYEjf28DpBN0+nbaTc242cB7BsNfrytm32k2uHYAWOo4H/cvM\nvuacuwI4iuDvYZGZ/fcA57oOmEfe/j4EttfJBPv938zst+FySYLRK45gUMIvzOzWQuvsp1yF6sMZ\nYYay7F+RKeQiIlJYVLpWRESkGyrkIiIRp0IuIhJxKuQiIhGnQi4iEnEq5CIiEadCLiIScf8fQsFw\nZbei13AAAAAASUVORK5CYII=\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "metadata": {
        "id": "tkToA3CjBxJe",
        "colab_type": "code",
        "outputId": "e75209fc-0db4-4637-d133-ce044a4e5efe",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 282
        }
      },
      "cell_type": "code",
      "source": [
        "plt.plot(history['loss_train'],'b')\n",
        "plt.plot(history['loss_val'],'r')"
      ],
      "execution_count": 73,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[<matplotlib.lines.Line2D at 0x7f651a9e6780>]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 73
        },
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXIAAAD4CAYAAADxeG0DAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJzt3XecVNX9//HXvdO20heIXYpHERtY\nEwuW2HsNsZdEE42aRL9q1NgVNXaTGH8aC2KvYMNYEUURe8HjWhACCisssH125t7fH3dmtsMubLvk\n/Xw89rFT7tz5zJ277z1z5txzHd/3ERGR8HJ7ugAREVk9CnIRkZBTkIuIhJyCXEQk5BTkIiIhF+3u\nJywrq1jlYTL9+xdQXl7dmeV0mt5am+rqmN5aF/Te2lRXx6xqXSUlxU5b94WqRR6NRnq6hDb11tpU\nV8f01rqg99amujqmK+oKVZCLiEhLCnIRkZBTkIuIhJyCXEQk5BTkIiIhpyAXEQk5BbmISMiFJsjd\n/86DCy6AqqqeLkVEpFcJTZAnpjwDEyYQf+etni5FRKRXCU2Qkz0BRn2qZ+sQEellwhPkkUypntez\ndYiI9DKhCfJ584P5CZYuUZCLiDQWmiD/dk4wUePc73SOURGRxkIT5ESCFrmfVotcRKSx0AS5k+kj\n99NqkYuINBaaIM992ZlO92wdIiK9THiC3FXXiohIa8IT5Bp+KCLSqtAEueNm+shT6loREWksNEFO\nVC1yEZHWhCbIcy1yBbmISBOhCXIyQe5o1IqISBPhCfLcAUEaRy4i0lhogtyJOIC6VkREmgtRkKtr\nRUSkNaEJ8mzXikatiIg0FZogz821oiAXEWkidEGODggSEWki2p6FjDHXATtllr/GWvtko/v2AK4G\n0sDz1toruqJQJ3dAkEatiIg0ttIWuTFmV2C0tXYHYG/g5maL3AocBvwC2NMYM6rTqwTNtSIi0ob2\ndK1MA47IXF4KFBpjIgDGmGHAEmvtPGutBzwP7N4lhWZHrXjqWhERaWylXSvW2jRQlbl6MkH3STZN\nhwJljRZfBAxf0fr69y8gGo10uNCivoUAxCIuJSXFHX58d1BdHaO6Oq631qa6Oqaz62pXHzmAMeYg\ngiDfcwWLOStbT3l5dXufsomqmiQAydokZWUVq7SOrlRSUqy6OkB1dVxvrU11dcyq1rWi8G/vl517\nARcCe1trlzW6awFBqzxr7cxtnc5R14qISKva82VnX+B6YH9r7ZLG91lr5wB9jDEbGGOiwP7AS11R\nqJPtjtGoFRGRJtrTIj8KGAQ8aozJ3vYq8Km19ingd8BDmdsfsdZ+1elV0mgcuUatiIg00Z4vO+8E\n7lzB/dOAHTqzqNZkJ81S14qISFPhObIzqrlWRERaE54gV9eKiEirQhPkblRdKyIirQlNkGsaWxGR\n1oUmyN3spFm+glxEpLHQBbmjFrmISBOhCXKNWhERaV1ogjz3Zae6VkREmghNkOPq5MsiIq0JTZC7\nsaBrRS1yEZGmwhPkUR0QJCLSmtAFueOra0VEpLHQBLmj4YciIq0KTZA3tMgV5CIijYUmyHWGIBGR\n1oUmyCNxtchFRFoTmiDPTWPr61RvIiKNhSbII7GgVFddKyIiTYQmyLPT2KprRUSkqdAEuaaxFRFp\nXWiCPNe1ogOCRESaCE2Qaxy5iEjrQhPkkaiDh6MgFxFpJjRB7jiQJqKuFRGRZkIT5AAeruZaERFp\nJnxBjoJcRKSxUAV50LWiIBcRaSxUQe7h6stOEZFmFOQiIiEXriB3NGpFRKS5cAU5rvrIRUSaCV2Q\na9SKiEhT4Qpyda2IiLQQriBX14qISAvR9ixkjBkNPAPcZK29vdl9c4B5QLapfLS1dn4n1pjjO+pa\nERFpbqVBbowpBG4DXlnBYvtYays7rao2eJprRUSkhfZ0rdQB+wILuriWlfIcF1ctchGRJhy/nScz\nNsZcCvzURtfKdGCDzO8LrLVtrjSVSvvRaGSVip0TH0leupqh6S7puRER6c2ctu5oVx/5SvwVeBFY\nAjwNHAY83tbC5eXVq/xEHhFc0pSVVazyOrpKSUmx6uoA1dVxvbU21dUxq1pXSUlxm/etdpBba+/P\nXjbGPA9sxgqCfHV4jg7RFxFpbrWGHxpj+hpjphpj4pmbdgE+W/2yWuerj1xEpIX2jFoZC9xA0Ade\nb4w5HJgMfGetfSrTCn/HGFMDfEgXtcYhOCAoQlpRLiLSyEqD3Fr7PjBuBfffAtzSiTW1ydfshyIi\nLYTqyE51rYiItBSqIM92rYiISINQBbla5CIiLSnIRURCLlRB7rnqWhERaS5UQe47LhG1yEVEmghd\nkAP4XvvmhxER+V8QsiAPJttKJ9W9IiKSFbIgD8r1UupeERHJUpCLiIRcuILcDbpWvHp1rYiIZIUr\nyLNfdqbVIhcRyQpXkLvqWhERaS5cQe6oa0VEpLlQBTm5FrnGkYuIZIUqyBtGrSjIRUSywhXkmVEr\nfkpdKyIiWSELcn3ZKSLSXKiCHAW5iEgLoQry7KgVda2IiDQIVZBnW+Q6IEhEpEGoglx95CIiLYUs\nyLNdKwpyEZGsUAW5oxa5iEgLoQpyX33kIiIthCrIyU1jqyAXEckKWZBnyvUU5CIiWaEMcvWRi4g0\nCFWQa64VEZGWQhXkRPRlp4hIc+EK8tyoFU1jKyKSFbIgD7pWSKtrRUQkK2RBri87RUSaC2WQq49c\nRKRBuII8olErIiLNRduzkDFmNPAMcJO19vZm9+0BXA2kgeettVd0epVZER0QJCLS3Epb5MaYQuA2\n4JU2FrkVOAz4BbCnMWZU55XXlKOuFRGRFtrTtVIH7AssaH6HMWYYsMRaO89a6wHPA7t3bomNqGtF\nRKSFlXatWGtTQMoY09rdQ4GyRtcXAcNXtL7+/QuIRiMdqTFndqZrJT8Rp6SkeJXW0ZV6Y02gujqq\nt9YFvbc21dUxnV1Xu/rIO8BZ2QLl5dWrvvJM10pVRQ1lZRWrvJ6uUFJS3OtqAtXVUb21Lui9tamu\njlnVulYU/qs7amUBQas8a21a6YLpNJmuFcdT14qISNZqBbm1dg7QxxizgTEmCuwPvNQZhbUqd0CQ\nDtEXEclaadeKMWYscAOwAVBvjDkcmAx8Z619Cvgd8FBm8UestV91Ua04Gn4oItJCe77sfB8Yt4L7\npwE7dGJNbdOoFRGRFkJ1ZKda5CIiLYUyyHVAkIhIg1AFebZrBQW5iEhOqIJcLXIRkZZCFeS5Frn6\nyEVEckIV5E40U67OECQikhOuIHc1akVEpLlwBXlUXSsiIs2FK8izX3bqgCARkZxQBrla5CIiDcIV\n5JmuFd/TpFkiIlnhCvJMi1zT2IqINAhlkOvIThGRBuEKco1aERFpIVRB7mYPCFLXiohITqiCXKNW\nRERaCleQR7Pn7FSQi4hkhSvII5prRUSkuVAFeUMfucaRi4hkhSrINWpFRKSlcAW5DggSEWkhVEHe\n0LWiFrmISFaoglxdKyIiLYUqyLMtcnWtiIg0CGWQq0UuItIgXEEeyxwQ5CvIRUSyQhXkDaNWFOQi\nIlmhCnJ1rYiItBSuIM90raCuFRGRnHAFuUatiIi0ENIgV4tcRCQrVEGem8ZWXSsiIjmhCvJITF0r\nIiLNhSrIc/ORq0UuIpITqiCPxNW1IiLSXLQ9CxljbgK2B3zgLGvte43umwPMA7L9HUdba+d3bpmB\nbIvc1ZedIiI5Kw1yY8wuwEhr7Q7GmE2AfwM7NFtsH2ttZVcU2Fi2j1xdKyIiDdrTtbI78DSAtXY2\n0N8Y06dLq2pDbq4VtchFRHLa07UyFHi/0fWyzG3LG912hzFmA2A6cIG1ts2TavbvX0A0O694B6XK\nKwCIOB4lJcWrtI6u1BtrAtXVUb21Lui9tamujunsutrVR96M0+z6X4EXgSUELffDgMfbenB5efUq\nPGVgYF7mA0Q6TVlZxSqvpyuUlBT3uppAdXVUb60Lem9tqqtjVrWuFYV/e4J8AUELPGst4IfsFWvt\n/dnLxpjngc1YQZCvjtwZgtRHLiKS054+8peAwwGMMWOABdbaisz1vsaYqcaYeGbZXYDPuqRSGo9a\n0QFBIiJZK22RW2vfNsa8b4x5G/CA040xJwDLrLVPZVrh7xhjaoAP6aLWOABu5shO1CIXEclqVx+5\ntfb8Zjd93Oi+W4BbOrOoNkV0QJCISHOhOrITJ/ie1fXVtSIikhWuIAfSuGqRi4g0EsIgjyjIRUQa\nCV2Qe7gatSIi0kjogtx3XFL1bR44KiLyPyd8Qe5G8NMey5b1dCUiIr1D6IIc1yVCmrlzw1e6iPwP\nSCaJvvcusWmvQyrVLU+5KnOt9Cgn4hKpTzNnjstmm+lLTxHpAN/HWbQICvLxi1ufxNWprMBZsgRv\n7XWCY1d8H5JJiEZzx7JkRWfNJPbBLLySwURmf0Hs3RnEPnwfp7YWgPTP1qL6j+dSe9yJuQMau0Lo\ngjzZr4S1f5zPE983n7tLRGQFqqvpe9x44tNeAyC52x5UTLgBb4MNyb/1Rgpu+htOOpULYT+RwM/P\nx6msxEml8GMx0hsOo+qSK0j+cm8i335N3yMOxq1qOBWD7zikR42mfrvtIe2RePwRiv/vj+Tf+Q/8\nvv2oPvNPcNyvOv2lhS7I64dvxKAfn2OxXQL0zikqRaT7RN+ZQf79/yZlNqbu4MPw1t8guCOdJvLZ\npyReeJbI93OIfFNK7P1Z1I8ZC45D/NWXGbDTtiTH7UZi6gt4gwaRWmdd/AED8fr1I/LNNzh1tfiF\nRfhFRTiVlUQ/+4Q+xxxF7dHHEfvgfdyqSqrOOR+/Xz9SIzYitfU2+H365mqrPuc8Ci88j8SLz0F0\nPu4PC7pmG3TJWrtQdNMR8BY4X5UCY3q6HJFewVm+jOLTf0vdvgdQN/6Yzlux7+d+CidcSfzll6j9\n1a+p33IsXkkJ3rDhHVpdbMZbRL74HL9vX0ilcKqrcSoriXz/HW5ZGX5hIXWHHk7yl3u3fHBtLU5V\nVdBKrq0hUlpK3tOPk3fPXbmTzRTcehPLH3iEvEn3w5SnGVBT02QVdXvvx/K77oNYjMTTT1B45aVB\niJcMpvy5/+BtsOEK649+/CF9Tjia/AfuA6Bm/DFU/99f2lzeGzKUirvuo/GktV3R/AxdkEc2GQlA\n4byvePbZbXFd2Hff7vlCQaS3KrzkQhJTXyD++qvUb/9zvA2HrfK6Eg89QP7Ee3EXLcRdtBBcl/Ra\naxP9uhSAoosvyC1bdd6FcM0VLdaRf8sNuMuWkdxlVwpuvI7IvLl4AwYS++SjlT5/3hOPktxxZ3Aj\nOOVLcJcsxi1fglPd+rkM0uutT8X1NxMttRRefAH9DtonuGPkSGrHbENy192p32osTipFesTI3FQf\ndYccTt2+B5B4bjL1W41daYgDpLbYiiVvv0909ue4CxeS3P2XK31MdwhdkKdGbATAoMVfceqpefTr\n5yvIpfeqriYx+Sn8omLqd9wJv1//lsv4PjgOkc8/I/+uO6g74CBwXBLPPEnN6WfhJxIU/eVc+Pn2\nRHbbB2/AQNyKZeD5pNdZl7xHHyJ/0v14AwfiLl5M0V/OZfn9D0MkglNZkfuo7yxbSsG1VxH57zzS\nGwwjveEw6rfelvRmm+dKib/0AsVnnw6uizdkKKmNN8GpqiJa+hV1u/+SyquuI/GfF3HLykg89TiF\n114Fr0ylaPOtiHz3LfXb7UB62HCKrroMgILbbwbAGzCA2H/nkdxtD2oPOxKnuho/Hof8fPz8AtLr\nrY83dCiRud9TdN6fiE+fFmyagkK8AQNIDR+J339A0MVRU42fl4+31lokx+1OcpddIS+P+l13xy8s\nouj8P1Nz3IkU/P1WKpbWrvj9SSSoO/SIjr2n+fmkxmzdscd0Mcf3u/fgmrKyilV+wpKSYn76cg6D\nNtmQyRzAQUwGYO7cCvLyOq3EVa5tTTobSVdbk+tyF/5I3n3/JvLtN8TffAO3bBEAfjxOze/+gB+P\nE5v5Dl7//kRLS4l89SXJXXcnPv1NnOqqJutKr7cBfnEx0c8/XeFz+nl5lD/3MkWXXkj8zTdIDx6C\nk07hLl5M3f4HkR42nMRjDxNppY+2fsutSI/YCKe6mvjrr4KXZumUqaS22KphoZoayM9v8jinrIzi\ns39P4tWXId1wtLXvuhCPU3X+xUQ/+Yjao35N/bjdoKoKiopWvgF9H+enn/D79IFEYuXLt3hB9RCL\nrXH7WElJcZsjPELXIvcHDqQybyCm1uZuW7DAYdgwHe35P8vzyL/zH+Tf9S8qJ/yN5B575e5yKitI\nPPEYyT32DIaTAbHXXyX6+WfU/PZ3EIut3nNXVUFBAe5/51Hwj1uJlJYSmzkjN/LBKyqm+qw/4ycS\n5D1wHwW33NDk4X48jrf2OiReehG/oICKa64n9t5M8D38fv3Jv+cuAGqOPo78cTtR88Z03GXL8Pr0\nwUmniXxdSv1WY6g5+VS8YcNZ/q97KLj5evIefhC/oIDUqNEknn0meK5YjKrzLqTmuJOIzJ1D5OtS\nElOeJv6fqcQ++jCod8AAKq6/uWmIQ4sQB/BLSlg+6TFKYmnK3/kAb/AQik//LfF33qbywkuoOfX0\npg9oT4gDOA5+SUn7lm3N6r6nIRS6FnlZWQX+L/ZkQOl7bL1JBR/PzufJJ6vZcceenX9lTfvv39U6\nXJfnEZs+DWf5crwhQ8B1cX/4gegnH5F44Vmi9stgsZLBLJk+E79ff+LPTqboovOI/LCA9DrrUnnd\njeRNvI/EC88CUHvgIaTGbE1s5juk11uf2IfvE/vyC2r2O5Ca359JeiPTpARn+TIi38+B6hpin3xI\n/LkpxN+eTnrIUNzly3AyX6ylNhxGzelnkdz9l3iDhzQES1UVeQ9Pwi8sJLn3vjjV1Xh9+0FBAdFP\nPsLr07dp37bvU3jx+biLFlJx278oWWdQ+7dZo7/r2DtvQ10dqbFbtzp22qmswFm6FD8Wxx88ONeH\n3F5N3stkkujsz0ltvmWH19PZ1ph9v+FxbW7QUAZ50Vmnk//QRO497yNOvHYLbr21hl/9qmf7yde0\nnaartVlXOg2RCLG3pxOb9hq1x59M7N0ZFF59OZE537W6Lj+RoG6/A/HWXoeC226ifuzW+Hn5xN96\nEz8eJ7nr7iSmvpBbvn67HcD3ic18p+l6HAdn8GBYuBA/EqHuiF/h9R+An5+HU1tH3n3/bjJmGKB+\nzFgi8+bhR6NUXXAxdQceAgUFq7+BWhG697KHrWl1rVFdK0DwzTMwsu4zYAsWLNDh+r1RbMZbRD7/\nFHfpUuIvT8Xv24+KG28L+nvnzyE2clPSI0biDR6Cu/BHiv/wO2IzppPadHTuo37BLTfipNP4iQQ1\nvz6WtNkk6HP2fbyBg0iNGkX9tjsEH9tTKaKzZhKf8RYAyZ12ofLaG0mPGEn+HbcTm/Y6NaecSv2u\ne+BUVVJ4yUV4665L7aFH4C5YgLfuugzc3LBs4iMUXn4xeQ9PavJ6vJLB1Bw1Hj+/gNTGm1D/8x3x\n1l0v92WlSE8JZZDX/2JHAEZ+Phk4mvnz9UfUEbEZb+H+sIC6vfaFwsJVW4nnEZvxFtGPP6J+2+0g\nGiX66SdEP/0Yp6oK98cfib/5em5x33VxPI8B222Jk0wC0K+V1abXXY/YRx+S2mQUtUf+mvy7/0V6\nxEgqrr1x5UPqolGWPfUc7ry5EIkEfeKZgK057QxqTjujoZ6iYipvaDhDobfe+sEF1yW57/4k99iT\n6AfvQzwWjHVevpzkuN1ab20rxKWHhTLIU1uNJb3Bhgya/iwFVDF/foLiP5yGW7aIZQ890eIPq7IS\nDjiggN//PskRR/yPDVVcuhT32zm5AzfcH3+g7/jDgv7ZomLqDjmMmhNOIb3Z5kTfmUF8xnTSGw7D\nnTePyJxvcaoqqd95V5I7jyPx7DO48+fj/vgDsZnvtDoCorHkjjtTe+wJ+AWF1G+7HYknHqXo0ouo\n22sfEiccR/XbM3G/n4O7+CfwPOoOOZza40/CqViOX1QMrkvN6Wd27PW6bsORfasjHie1/Q6rvx6R\nbhDKIMdxqD30CApvvI7x+U+TKh1M3qsPAhD54nPSm45usvgXX7h8/nmEKVOiuSB3li8j+tmn1P98\nx24vH4IjxAr+NoHKCTfkRlO0h/vtNySmvoBTVUl09hc4S8up3/7nOEvLcRf/RN1Bh5H85V64C3+k\nz29OgFkzGQgsv+Nu6g49goLrrsaprqZu3wOIfvwh+RPvJe/BidSOP4a8hx7ASbf80jjvycdb3OYN\nGkTtkeNJ7rQLsVnvgeuQ2nxLUpttjtevP6TTwQEWjf6p1p5yGrXHn5wbGla1+36tvsbGhziLyMqF\nM8iBusOOpPDG6/ijfyOp+Q2zICaefYbqTUdDMkn+HX/HLy6m1tsD2IzS0oaZy4rOP4e8xx+h/IVX\nSI3dpnuL9zyK/nxWcJSb4wQHbwD4PvFnnyH//nuoOe4kkgccFNxeXx9MhxmP0/fEo4nO/qLJ6uJv\nvpG7nPfk43gDB4Lj4v5UBjvvjD9rVnBASSpF3oMTSZmNg8OUHYf4yy9R/MczyH/gPrw+fam44hrc\nZUtJr70O6Y02Btcl7967iH75JXX7H0hqzFi8AQPx1lm34Qi5o37d/tf+Pzg0TKSrhTbI0yM3ou6X\ne7Hpf6YC8FrhvuxQ9QqJZydTfd6FFNx6I4XXXQ3AwZEYv+BVKr7rS+Thd/F225XEM08CkHj6iY4F\neX198OVWPN7+x/g++XfdQezVl3ErK0mN3ozYJx/huy6JF5+n8PK/4ixbSmzWe0Rnfw5A/I3XSO60\nC95aaxN/eSo4DjUnnEJ09hfU7bM/NSecTHrESPyiImLvzMAbMBAK8smbdD+JKc/g/FRG5aVXUfTX\nC6i66lqKLr6APmecih+JUHnFhGBKTiC51z6U/+cN8u+4ndqjjiY9erMW5VdddV37X6uIdLtQDj/M\nSae5/9g38V9+gxv4M3dwGgcxmeX/+H8U//EMvP4DqDnjLPIvvpClfl8KqSKPOlKjRhP94rNgFWut\nzdJnX8pMBnQ02UNEneXL8KMxKCgg74H7cKoqqT1yPH0PO5DIjwtYfvud1O+2R1BHMknJWgMo+ykz\nNK22lvi014jM/oL6XXcn8s3X9Dn1JCAzxM338eNxlt87iT7HjcfJTD7vJxIkx+1GzYm/ofD6a4i9\n/x4AXv/+uOXlwTLRKEtmfLDifuB0Gqe8HH/QoGCb/biU4lNPwqmrpeovl5DeZNSqvgWdZk0bGtYd\nemttqqtjNI68lQ1w881xrr46OIz3GCYykeNy9y27+36SBxzM1H3+xTHvn8tPDCS/X5zCpT/g5+eT\nHLc7iReezc1RUbfvAdSceAr5d/+L+MsvkV5vfWrO/FMw9wTg9e2Hu2xpbv11e+2DX1RM4pkncYqK\nqNtmO2pP+g2FV16WO6Taz8sLZmurqaH8len4kQgFt99MaszW1B5zPLE3XiMy/7/Ub7YF6Y03adL1\n4JQvwV2wgPTIjci/618UXXohNSf9hsoJTY8O7Og26w1UV8f11tpUV8coyFvZAE89FeXUU/PZfvsU\n77/jcc+Ymzh09Fek112XmjP/BI7DMUfnwX9e5RM258KT5vL7x/YMpuIctxt9jz4SgPSQoUQW/phb\nb3q99YnM/R4IDqNObTWW2LszqD3oUGpOP5Oi8/5E7MMPAEgNG040GoGvvso9vvbI8dSP3Yaiyy7C\nqa6m8qJLg3pWQ+SbUtIbDGtxlpIVWdN25q7WW+uC3lub6uoYHRDUiv32S/G3v9Vy8MH1bLFFEZdX\nnsOef2s63eWiMpePCObfmFYxiKM+Kw0m40mlqN9yK1JbjKHqr5dR/LtTIJ6g+g9nk9pyDIUXnUfB\n/7uDykuvpPaEU4jOeo/U2K0hFmPp1NeJfPYpbmUF9dvtQMngPix97BkKbruZun33p/ak34DjUL/1\ntsSnT6PmlFNX+7Wmh49c7XWIyJon9C3yxg4/PJ9p06JYW0EsBhddlODEE+s57rh8XBfKyhw23dRj\n6tQg6D0Pbr89zoYbehxwQOvjy52FC/GHDFnt2nqK6uqY3loX9N7aVFfHqEW+Ettsk2batCizZkWY\nO9flwQfjeJ5DWZnDllt69OnjU1rq5o6ovv32OFdeGfSvH3tskuuvr2txftT2hLiISE9aoyYp+fnP\ng4NZHnssxqOPBl8aTpsWIZVyGDzYwxiPykqHe++N8eSTUa65Js7QoR6bbppm4sQ406a1v+9ZRKS3\nWKNa5DvumGaLLdI8/XTDyI/shFqDB/uccEI906dHOO+8YIhhPO5z5521LF8OxxxTwKxZEcaN69np\ncEVEOmqNapE7Dlx0UV3u+lprNRzxOWSIz6hRHlOmVDNmTJpDD63n9der2H77NFtuGSz34YdqkYtI\n+KxRQQ6wyy5p9twzxdpre5xxRjJ3++DBwXesw4f7vPhiNXfcUcuIEX7uvnXX9fjgA5eFCx1uuilO\ns5Nvi4j0WmtckAPce28NM2ZUsc02Dd0kgwd7K3gEjBmTZvFil5NPzuOaaxJMnBhj/nyHc89N8MMP\nnT9N6cKFDjffHKeNE4OLiLTbGhnk0WhwpP3GG3tEow2t7hXZaqsg9GfODL42mDQpxmWXJbjvvjhn\nnJGH1+j/wNdfOxx8cD4vvth2V0x5ORx8cD4339z6nCyXXprg6qsT3H13B+ZsaaayEj79dM17C+vr\ne7oCkXBp15edxpibgO0BHzjLWvteo/v2AK4G0sDz1toruqLQVZFIgDEen38eYciQFQf5mDENST1o\nkMfs2RFmzw6C+s03o5x/foKNN/bYf/8UZ52Vz3vvRfjggwhPP13d5LFZV16Z4O23o7z9dpRo1Gfr\nrT1GjUrTpw/Mnevw9NPBpr/rrhinnZbs8KSAngfjx+fz7rtRLrqojpISj5kzI5x5ZpKSEp8lSxzW\nWy98J6T+5huHAw8sYM89U9x4Y12vO2fDsmXw8MMxDjkktdLGgUh3WWmQG2N2AUZaa3cwxmwC/Bto\nPOP+rcBewHzgDWPME9baL1pZVY84/vh6XnvNW2mQb755mnjcZ/Bgn2uuqeXYY4MzwVx/fS1XX53g\n3nuDlvMll/jU1TlssUWaTz9wNIkbAAALM0lEQVR1OfDAArbbLs1228HcuXlMnhxl/fU9rI0wcmSa\n8nKHyy8PRsmUlHhceWUdr7wSJZ12GDbM49tvXS65JEFlpcOPPzrk5fkY47HjjmkcB954I0L//j4j\nRngMH+4zYIBPYaHPpEkx3n03ePuyY+EBnnwyhuNAdbXD8ccn+ec/oa4OvvvOpb4eBg3yGTjQJxoN\nWvQVFU7mJ7icnw+jRqUpLXUpL3fYeus0/fuveBtnjynz/aaXsxyn6U9bkkk47bR8yspcJk2Ks9VW\nHuPH1+M44LqtPz6ZhNmzg9e2wQY+SzNT4ay/vk8sFtSRSgU/6XTwO/uJLdps76+qavg0EIsF2+eD\nDyLE48EntmQSjjqqgA8+iDBxYpqnnqqhpMRv8hw1NeD7Dv37+y2OSWiPZBK++solnYbhwz0KCla+\n3Xw/+MeeTje8xtpah7o6KC93KC11GTDAZ4cd0iQSLR+/YIHDNdckWLYMLrggySabrLgbcnUtXw7f\nfusyaJDPz37m597bjvr+e4c774wTj8OppyYZOtRn4UKHr75yGT7co6Sk7cd6Hnz7rUNZmcuoUWn6\ndsIU+L4PDzwQY8KEOHvtleKyy+ooLl799bbHSo/sNMZcDsy11t6Vuf4lsK21drkxZhhwv7V2x8x9\nFwCV1trb2lpfVx7ZubrefjvCwIFBaO6ySwH9+vlMmVLDokUOM2dGmDPH5aab4sTjPm+8Uc1bb0W4\n/fY4n33W0MWy9toeixY5pNMwZUo1xcXw4IMxksngTU4mndxyjz5aw447FuD7q9bs7NfP56GHqrn0\n0gTDh3tss43HDTcEO3Yk4jeZf311RKN+i5Be1ZoBHMfHcZzMb3JB7fuQTDrss089M2ZEWbq05XNk\nH+O6wU8QXi2Xc91gudbua/y6stPWZJ87+xzRKNTXNzw2EgkCp77eYeTINKWlEVzXb3NbRCJ+k09Z\njf/M2roMQQi3tW2D1900+Np6/a1pXlN2HXV14HkN60gkOvon6hB8WG9fINfWtv6+RiLBNELtDfW6\nuoZt5brBa6ura3hwXl6w3tak0w3vN0As1nK55nW0Vlfj23y/6fPHYn6TxoLjwMUX13H++Xk9cmTn\nUOD9RtfLMrctz/wua3TfImD4ilbWv39BMMHUKiop6bp/cQcd1HD5k0+CDR+PFzN4MIzOnHTo7LOD\nP+YhQ4oYPRpOPRWWLIHZs4P7d9jBZfFiWLwYNt44OB/mTjs1PPahh4Id7IADXLbaqpC77oL584Pn\nHjEiaBV+8AFMnRr8Ue+3H1RXg7VQWhp8tK+sDG475xyHvfcuZO+9G+o+++zgd3U1XHRR8Do8D4YP\nh/x8WLQo+EmnoW9f6NOn4adv3+C1fPxxsPzgwTBjBlRVZQOuYcdt7+XsP4CmP06mFem0uG/ECLjj\njhgffgjXXpv9Yw1eg+cFjw1+B9cjEdh882CbfvstDBoUvLavvw7WHY0GretoNPiJRBpazjU1Tu67\nD9+H/v2DbbBwYdCaLSyEsWODZWfNckilYOedYcKECBMmwPPPO7l1Zn8XFgbrWrgwWL6xxn/0bYVE\nNAqjRgW/S0uDTwiNX2/29Wfrbvzc2Z9oNHiv8/OhuBiMge+/hxkzggZG9vVmxeNwyilQUgI33xx8\nMus4p8U/pbb07QsbbwxlZbBwYfbThJP7RNFeBQXwm98E+/r99wfv2ZAhsOmm8OWX8N//gtPGfwXH\ngZEj4Wc/C/b3ysqG5Zq/jhX9421+ffhwuOoqmDQJpkxp+tyuC5tskv103rk51p4W+Z3Ac9baZzLX\npwMnWWu/Msb8HDjXWntI5r5TgGHW2r+0tb7e3CJfHb21NtXVMb21Lui9tamujumKuVba04u3gKDl\nnbUW8EMb962duU1ERLpJe4L8JeBwAGPMGGCBtbYCwFo7B+hjjNnAGBMF9s8sLyIi3WSlfeTW2reN\nMe8bY94GPOB0Y8wJwDJr7VPA74CHMos/Yq39qo1ViYhIF2jXOHJr7fnNbvq40X3TaDocUUREutGa\nd1igiMj/GAW5iEjIKchFREJOQS4iEnLdfvJlERHpXGqRi4iEnIJcRCTkFOQiIiGnIBcRCTkFuYhI\nyCnIRURCTkEuIhJy7Zo0qzdY0Qmge6ie64CdCLbhNcCBwFhgcWaR6621z3VzTeOAx4DPMzd9ClwH\nTAQiBPPIH2utrevmuk4Gjm1009bALKAQqMrc9mdr7fvNH9uFNY0GngFustbeboxZl1a2kzHmaOBs\ngpk/77TW3t0Ddd0DxIB64Bhr7Y/GmHrgrUYP3d1a24Hz66x2XffSyv7eC7bXY0D2bJ0DgHcITg7/\nKQ1nOiuz1h7RxXU1z4f36ML9KxRB3o4TQHd3PbsCozP1DAQ+BF4FLrDWPttTdWW8Ya09PHvFGHMP\n8Hdr7WPGmKuBk4B/dmdBmZ3z7kw9uwBHApsCJ1prP+vOWjI1FAK3Aa80uvlymm0nY8z9wF+BbYEk\n8J4x5ilr7ZJurOtKgj/wR40xpwN/Av6PYBrpcV1RRzvrgmb7e2a5Ht1ejQPaGPNv4K6Gu7pte7WW\nD6/QhftXWLpWdgeeBrDWzgb6G2P69GA904DsDrOUoGXZOWc67nzjgMmZy1OAPXquFCDYca/o4Rrq\ngH1pejarcbTcTtsB71lrl1lrawhawL/o5rp+DzyRuVwGDOzC529La3W1pjdsLwCMMQboZ62d2YXP\n35bW8mEcXbh/haJFzopPAN3tMh9hs10CJwPPA2ngDGPMnwhOQn2GtfanHihvlDFmMsHHysuAwkZd\nKYuAn/VATQAYY7YB5mW6BgAuN8YMAmYDZ2d25i5nrU0BqUwNWa1tp9ZOLt5l26+1uqy1VQDGmAhw\nOsEnB4A8Y8yDwPrAE9baG7uzrowm+zu9YHs1chZBaz1rqDHmcYJTVf7dWjupC+tqLR/26sr9Kywt\n8uZW5TTfnc4YcxDBG3UGQf/X+dba3YCPgEt7oKRSgvA+CDieoDuj8T/rnt5upwD3Zi7fQnDi7p3J\nnHmqp4pqRVvbqUe2XybEJwKvWmuz3QjnAL8F9gSONsZs3c1ltWd/76ntFQd2tNa+lrlpMXAxMJ7g\nu6wrjDFd3qBplg+Ndfr+FZYW+YpOAN0jjDF7ARcCe1trl9G0/3Ay3dwPDWCtnQ88krn6jTHmR2Ab\nY0x+prXb0yfHHgf8ASBzmsCsKcBRPVFQI5WtbKfWTi7+Tg/Udg9Qaq29LHuDtfaO7GVjzCvAZgRf\nIHeLRv9QoGF/f5zesb12AXJdKplzDN+TufqTMWYWsDFdmCHN88EY06X7V1ha5G2eALonGGP6AtcD\n+2e/mDDGPGGMGZZZZBzQE1/iHW2MOSdzeSgwhGAHPiyzyGHAi91dV6aetYBKa23SGOMYY142xvTL\n3D2OHthezbxMy+30LsE/wn7GmCKC/ss3u7OozKiGpLX2kka3GWPMg5ntGM3U9XmbK+maulrb33t8\ne2VsQ6PTURpjdjXG3Ji5XAhsCXTZuYVbywe6eP8KzTS2xpgJQO5juLX245U8pCtr+S3BR8nGO8M9\nBB+hqoFKghEZi7q5rmLgQaAfECfoZvkQuB/IA77P1FXfnXVlahsLXGmt3Sdz/UjgPIK+xPnAydba\n6m6s5QZgA4IhffOBowm6fZpsJ2PM4cC5BMNeb+vKvtU26hoM1NLwfdAX1trfG2OuBXYj+HuYbK29\nqpvrug04n2b7ey/YXocS7PfTrbWPZJaLEoxeMQSDEv5prb2ntXV2Ul2t5cPxmRq6ZP8KTZCLiEjr\nwtK1IiIibVCQi4iEnIJcRCTkFOQiIiGnIBcRCTkFuYhIyCnIRURC7v8DdInj8UxkxvkAAAAASUVO\nRK5CYII=\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "metadata": {
        "id": "NJPdR4IFb-uc",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 282
        },
        "outputId": "efeac19f-1b6b-4429-8884-4e9502a018ee"
      },
      "cell_type": "code",
      "source": [
        "plt.plot(history['learning_rate'])"
      ],
      "execution_count": 74,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[<matplotlib.lines.Line2D at 0x7f651ab05710>]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 74
        },
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYMAAAD4CAYAAAAO9oqkAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJzt3Xt8XHWd//HXZCb3S5OmadOU0NLS\nfuiFiwUWCmqrrbAqq7sL617wgovrPpDdRf25yu/h46crKLrrruyyN9fHT3BFUVQeuPCzIgoqSLm0\nFYq25VNaWnpJ26RtmqRt7jO/P+akTEMmnUkzOZPM+/l48EjmnPM98z7DdD453+8534kkEglERKSw\nFYUdQEREwqdiICIiKgYiIqJiICIiqBiIiAgQCzvAWLS1dY35Eqi6ugra20+MZ5xxk6/ZlCs7ypW9\nfM021XI1NFRH0q0ruDODWCwadoS08jWbcmVHubKXr9kKKVfBFQMREXk9FQMREVExEBERFQMREUHF\nQEREUDEQERFUDEREhAIrBntbj/HtR14irmm7RUROUVDF4JktB/nuT52dLZ1hRxERySsFVQxm1pUD\n0HLoeMhJRETyS0EVg9n1FQC0HFYxEBFJVWDFoBKA/Yfzb+IpEZEwZTRrqZndCVwOJIBb3H19yro1\nwB3AILDW3W9P18bMVgBfBvqBXuB97t5mZtcDHwXiwNfc/evjdYCpqsqLqasuVTeRiMgwpz0zMLOV\nwEJ3XwHcCNw1bJO7gGuBK4GrzGzJKG0+Drzf3d8CPA38hZlVAp8B1gCrgI+Z2fQzPrI0mmdVc7ij\nh96+wVw9hYjIpJNJN9Fq4IcA7r4VqDOzGgAzmw8ccfc97h4H1gbbj9jG3f/I3V8xswgwB9gLXAas\nd/cOd+8GniJZWHKieVY1CeDAEXUViYgMyaSbqBHYmPK4LVjWGfxsS1nXCiwAZqRrY2a/S/JMYSvw\nLeBPRtjH7NEC1dVVjHk+7+aZVQB09Q3S0FA9pn3kUj5mAuXKlnJlL1+zFUqusXzTWdpvyhll3cnl\n7v6ImRnwJeBWYFcW+wc4o28eam5MvoC+8zDLzq4d835yoaGhmra2rrBjvI5yZUe5spev2aZartEK\nSCbdRC0k/6of0gTsT7NuTrBsxDZm9gcA7p4AHgDeOMo+cqJ5VvLF0CCyiMhrMikGjwLXAZjZcqDF\n3bsA3H0XUGNm88wsBlwTbJ+uzd+Z2UXBfi8DHHgWuNTMas2siuR4wZPjdHyvU1tVSmVZjBZdXioi\nctJpu4ncfZ2ZbTSzdSQv/bzZzG4AOtz9QeAm4DvB5ve7+zZg2/A2wfobgf8wswGgm+Slpd1mdivw\nE5KXoX7O3TvG8RhPEYlEmD2jklf2ddI/EKc4VlC3WoiIjCijMQN3v3XYok0p654AVmTQBnffAFwx\nwvIfAD/IJMt4aKqvYPveDg62n+CshqqJeloRkbxVkH8WNwV3ImvcQEQkqTCLwQxNSyEikqogi8Fs\nnRmIiJyiIIvB9JpSSkui7NfspSIiQIEWg0gkwuzpFRw4coLBeDzsOCIioSvIYgDJcYOBwQRtR3vC\njiIiErqCLQZzgkHkfW3HQk4iIhK+gi0GZwUT1u1pVTEQESncYhDcbLa3TYPIIiIFWwxqq0qoKi9m\nr84MREQKtxhEIhHOaqik9Wg3PX0DYccREQlVwRYDeG3cYJ+6ikSkwBV0MWgOxg326IoiESlwBV0M\nhs4MNG4gIoWuoItB04xKIhEVAxGRgi4GpcVRZtVVsKftOIlEIuw4IiKhKehiAMmuou7eAY509oYd\nRUQkNAVfDJobktNSaBBZRApZwRcDDSKLiKgYnLy8dK/ODESkgBV8MaifVkZZSVQT1olIQSv4YhCJ\nRDhrZhUHj3TTPzAYdhwRkVAUfDEAaJ5ZRTyRoOXQibCjiIiEQsWAZDEAePVgV8hJRETCEctkIzO7\nE7gcSAC3uPv6lHVrgDuAQWCtu9+ero2ZNQP3AMVAP/Bedz9gZv3AUylPudrdJ6zPZu6sagBePdAF\nF07Us4qI5I/TFgMzWwksdPcVZrYYuBtYkbLJXcDVwD7gl2b2ANCQps3nga+5+/fM7Gbg48AngQ53\nXzWOx5WVsxqqiBZF2HVAZwYiUpgy6SZaDfwQwN23AnVmVgNgZvOBI+6+x93jwNpg+3RtPgI8EOy3\nDagfx2MZs+JYEXMaKtnTeoyBwXjYcUREJlwm3USNwMaUx23Bss7gZ1vKulZgATBjpDbuvg3AzKLA\nzcBtwfoyM7sPmAs84O5fGS1QXV0FsVg0g+gja2ioft2y8+bVs/vgMXricE7j69dPlJGy5QPlyo5y\nZS9fsxVKrozGDIaJjGHdyeVBIbgXeNzdHwsWfwL4FsnxhSfM7Al335DuSdrbx37VT0NDNW1tr+8O\nmlVbBsDzWw9QVRzOuHq6bGFTruwoV/byNdtUyzVaAcmkGLSQPAMY0gTsT7NuTrCsb5Q29wAvu/vn\nhla6+1eHfjezx4DzgbTFIBdSB5HfdMFEPrOISPgy+RP4UeA6ADNbDrS4exeAu+8CasxsnpnFgGuC\n7UdsY2bXA33u/tmhnVvSfWYWCfZxJbB53I4wQ80zK4kWRZJXFImIFJjTnhm4+zoz22hm64A4cLOZ\n3UDyCqAHgZuA7wSb3x+MC2wb3iZYfzPJ8YFfBI+3uPtHzGwP8Fyw7UPu/tw4HV/GimNRmmYkB5EH\n43GiRboFQ0QKR0ZjBu5+67BFm1LWPcGpl5qma4O7X5Fm/5/KJEeuzW2sZk/rMfYfOnFyNlMRkUKg\nP39TzAuuItL9BiJSaFQMUsxtTLkTWUSkgKgYpGhuqKIoEtEcRSJScFQMUpQUR2maUcHu1i7i8UTY\ncUREJoyKwTBzG6vp64+z//DxsKOIiEwYFYNh5jXWALBzv7qKRKRwqBgMM78pWQxe2d8ZchIRkYmj\nYjBM88wqimNFvLKvI+woIiITRsVgmFi0KHnzWdsxevv0ncgiUhhUDEawoKmGRAJ2qqtIRAqEisEI\nFjRNA2BHi7qKRKQwqBiMYMGcZDF4pUVnBiJSGFQMRlBXXUpddSk7WjpJJHTzmYhMfSoGaSxoqqHz\neB+HO3rCjiIiknMqBmnMPzluoK4iEZn6VAzSWDAnefPZDt1vICIFQMUgjbmzqokWRXRmICIFQcUg\njZLiKGfPqmL3wS76B3TzmYhMbSoGo5jfNI3BeIJXDx4LO4qISE6pGIxiQTBp3fa9GjcQkalNxWAU\ni5prAdi252jISUREckvFYBTTa8qYMa2Ml/ceJa6bz0RkClMxOA1rruV4zwAtbfrmMxGZulQMTmOo\nq8jVVSQiU1gsk43M7E7gciAB3OLu61PWrQHuAAaBte5+e7o2ZtYM3AMUA/3Ae939gJldD3wUiANf\nc/evj9cBnqlFZ79WDFZffFbIaUREcuO0ZwZmthJY6O4rgBuBu4ZtchdwLXAlcJWZLRmlzedJftiv\nBB4EPm5mlcBngDXAKuBjZjb9jI9snMysLWdaVQnb9hzVpHUiMmVl0k20GvghgLtvBerMrAbAzOYD\nR9x9j7vHgbXB9unafAR4INhvG1APXAasd/cOd+8GniJZWPJCJBLBmmvpPN7HwfbusOOIiOREJt1E\njcDGlMdtwbLO4GdbyrpWYAEwY6Q27r4NwMyiwM3AbWn2MXu0QHV1FcRi0Qyij6yhoTqr7ZcvbuS5\nra20tPdwvs0a8/NmIttsE0W5sqNc2cvXbIWSK6Mxg2EiY1h3cnlQCO4FHnf3x8zsz7LYPwDt7SdO\nGzKdhoZq2tq6smrTVFcGwMYtB1i+IHc9WGPJNhGUKzvKlb18zTbVco1WQDLpJmoh+df7kCZgf5p1\nc4Jlo7W5B3jZ3T93mn3kjaYZlVSWxXTzmYhMWZkUg0eB6wDMbDnQ4u5dAO6+C6gxs3lmFgOuCbYf\nsU1w1VCfu382Zf/PApeaWa2ZVZEcL3hyXI5unBRFIixqruVwZw+HOjRuICJTz2m7idx9nZltNLN1\nJC/9vNnMbgA63P1B4CbgO8Hm9wfjAtuGtwnW3wyUmdkvgsdb3P0jZnYr8BOSl6F+zt3zbjKgRc21\nPP/yIbbtOcqMaeVhxxERGVcZjRm4+63DFm1KWfcEsCKDNrj7FWn2/wPgB5lkCct5Z9cBsPXVdq5Y\nNur4tojIpKM7kDPUPKuKqvJituxq1/0GIjLlqBhkqCgSYfHcOtq7ejlwZOxXM4mI5CMVgywsPSd5\nWenmnUdCTiIiMr5UDLKwZF5y3GDLrvaQk4iIjC8VgyzMmFbOzLpyXtrdzsBgPOw4IiLjRsUgS0vn\nTaenb5Cd+zvDjiIiMm5UDLK0ZF5y3EBdRSIylagYZGnx3FoiEdi8S4PIIjJ1qBhkqaKsmHNm1/DK\nvk66ewfCjiMiMi5UDMZgybzpxBMJfLcmrhORqUHFYAyWBpeY/nbn4ZCTiIiMDxWDMVgwZxrlpTFe\n3HFYU1OIyJSgYjAGsWgR58+fzqGOHloOHQ87jojIGVMxGKMLF8wAYNMOdRWJyOSnYjBGy+ZPJxKB\nTdsPhR1FROSMqRiMUXVFCQvmTGP7vg6OdfeHHUdE5IyoGJyBCxfUk0jAb15RV5GITG4qBmfg5LiB\nuopEZJJTMTgDcxoqqa8p5bevHNEspiIyqakYnIFIJMIF587gRO8AO/Z1hB1HRGTMVAzO0FBX0Qvq\nKhKRSUzF4AwtnltLWUmUjd6mu5FFZNJSMThDxbEoF547g0MdPew+eCzsOCIiY6JiMA4usQYANnhr\nyElERMYmlslGZnYncDmQAG5x9/Up69YAdwCDwFp3v320Nmb2N8A/AXXufixY1g88lfKUq9198AyP\nbcIsm19PSXER619q5Q/fPJ9IJBJ2JBGRrJy2GJjZSmChu68ws8XA3cCKlE3uAq4G9gG/NLMHgIaR\n2pjZ+4FZQMuwp+lw91VnfDQhKS2OcsGCGWx4qZW9bcdpnlkVdiQRkaxk0k20GvghgLtvBerMrAbA\nzOYDR9x9j7vHgbXB9unaPOjunyZ5tjClnOwqekldRSIy+WTSTdQIbEx53BYs6wx+tqWsawUWADNG\nauPu29I8R5mZ3QfMBR5w96+MFqiuroJYLJpB9JE1NFSPuW06b60p5+4fbeWFHYf58LUXjnk/ucg2\nHpQrO8qVvXzNVii5MhozGGa0DvF0607Xif4J4FskzxieMLMn3H1Duo3b20+cZnfpNTRU09bWNeb2\no1l6znSef/kQL2w9wJwZlVm3z2W2M6Fc2VGu7OVrtqmWa7QCkkk3UQvJM4AhTcD+NOvmBMtGa/M6\n7v5Vdz/m7seBx4DzM8iVdy45byYAG9VVJCKTTCbF4FHgOgAzWw60uHsXgLvvAmrMbJ6ZxYBrgu3T\nthnOku4zs0iwjyuBzWd2WOG46NwZxKJFPLv1oG5AE5FJ5bTdRO6+zsw2mtk6IA7cbGY3kLwC6EHg\nJuA7web3B+MC24a3ATCzTwNvI3nW8GMze9rdP2lme4Dngm0fcvfnxvcwJ0Z5aYyLFiavKtp1oItz\nZteEHUlEJCMZjRm4+63DFm1KWfcEp15qmq4N7v4F4AsjLP9UJjkmgyuWNrLhpVae3nxAxUBEJg3d\ngTzOls2fTlV5Mc9tOchgXNNai8jkoGIwzmLRIn5n8Uw6T/SzeWd72HFERDKiYpADK5YlL6R6evOB\nkJOIiGRGxSAH5s+uYVZdOc9va6O7dyDsOCIip6VikAORSIQVSxvpG4jz621tp28gIhIyFYMcuXzp\nLACe+k3ae+1ERPKGikGOzKyr4Lyza3lp91EOHBn79BkiIhNBxSCHVl40B4AnXhg+Y7eISH5RMcih\n5YsaqCov5le/2U//gO45EJH8pWKQQ8WxIq48v5Fj3f08/7IGkkUkf6kY5NhQV9Evnt8XchIRkfRU\nDHKscboGkkUk/6kYTAANJItIvlMxmADLFzVQXVHMky+20Ns3GHYcEZHXUTGYAMWxIt7yhjkc7xlg\n3W91E5qI5B8VgwnyluVnEYtGeHTDXuL6FjQRyTMqBhNkWmUJly2ZxcEjJ3hxx+Gw44iInELFYAK9\n7ZJmAH66fk/ISURETqViMIHOnlXN4rl1bH21nd0Hu8KOIyJykorBBLvqUp0diEj+UTGYYOcvqGd2\nfQXPbDnIoY7usOOIiAAqBhOuKBLhnSvmMhhP8ONndocdR0QEUDEIxWVLZjGztpwnX2yhvas37Dgi\nIsQy2cjM7gQuBxLALe6+PmXdGuAOYBBY6+63j9bGzP4G+Cegzt2PBcuuBz4KxIGvufvXx+fw8lO0\nqIh3rJjLN378Ej9+9lX+bM2isCOJSIE77ZmBma0EFrr7CuBG4K5hm9wFXAtcCVxlZkvStTGz9wOz\ngJaU/VcCnwHWAKuAj5nZ9DM8rrx3xbJG6mtK+eULLXQc7ws7jogUuEy6iVYDPwRw961AnZnVAJjZ\nfOCIu+9x9ziwNtg+XZsH3f3TJM8WhlwGrHf3DnfvBp4iWVimtFi0iHdcPpf+gTg/eU5jByISrkyK\nQSOQ+s0sbcGykda1ArPTtXH3kS6uT7ePKe+NFzRRV13K4xv3cvSYxg5EJDwZjRkMExnDutHaZL1t\nXV0FsVg0i12eqqGhesxtx9v1v3se//b9TTy6cR8Lz5mRV9lSKVd2lCt7+ZqtUHJlUgxaeO1MAKAJ\n2J9m3ZxgWd8obU63/znAM6MFam8f+5fENDRU09aWP3f/XnhOHY3TK3j0mVd595vnU5pN2Zwg+faa\nDVGu7ORrLsjfbFMt12gFJJNuokeB6wDMbDnQMtTd4+67gBozm2dmMeCaYPu0bUbwLHCpmdWaWRXJ\n8YInM8g1JUSLirh25QLiiQTfXLs17DgiUqBOe2bg7uvMbKOZrSN56efNZnYD0OHuDwI3Ad8JNr/f\n3bcB24a3ATCzTwNvI3km8GMze9rdP2lmtwI/ITmw/Dl37xjfw8xvyxfNYMGcGp7+zX7eelETC+ZM\nCzuSiBSYSGISzq3f1tY15tD5etq3bc9RvvTtX7PwrGncev1yIpH86S/K19dMubKTr7kgf7NNtVwN\nDdVpP1h0B3KeWNRcy2VLG3l5bwfPbj0YdhwRKTAqBnnkQ+9eRixaxPce305370DYcUSkgKgY5JHG\n+krecfnZHD3Wx8PrdoUdR0QKiIpBnnnH5XOZMa2Mn67fQ8uh42HHEZECoWKQZ0qKo/zJ6oUMxhN8\n+6fbmIwD/CIy+agY5KE3LJzB+fPr2fpqO7/6Tbp79URExo+KQR6KRCK8/2qjtCTKdx/bru88EJGc\nUzHIU/XTynjPW86lu3eAbz7ykrqLRCSnVAzy2MqLmjjv7Fo27TjMM1t074GI5I6KQR4rikS44R2L\nKSku4r6fblN3kYjkjIpBnptZW84fv+VcjvcM8LWHNhOPq7tIRMafisEksOoNc1i+qAHfc5QfPb0r\n7DgiMgWpGEwCkUiEG95+HtNrSvmfX+3i5b1Hw44kIlOMisEkUVVezId/bykJEnztoc10negLO5KI\nTCEqBpPIouZa3v3Gczjc2ctX/2czg/F42JFEZIpQMZhkrrliHm9YOIOtr7bz/Z/vCDuOiEwRKgaT\nTFEkwoeuWcLs+goeXb+HpzRdhYiMAxWDSai8NMZfX3sB5aUx/vsRZ/vegvqWUBHJARWDSapxegU3\nvXsp8XiCf/nBJvYf1nTXIjJ2KgaT2LL59Xzg7cbxngG+cv8mjh7THcoiMjYqBpPcmy5o4vffdA6H\nO3v45+9t4kSPvi5TRLKnYjAF/N4V81h1URO7W49x5/de0Pcni0jWVAymgEgkwnuvMlYsncWOlk7u\n/P4mevpUEEQkcyoGU0RRUYQb37mEy5bMYvveDv75+y+qIIhIxmKZbGRmdwKXAwngFndfn7JuDXAH\nMAisdffb07Uxs2bgXiAK7Afe5+69ZtYPPJXylKvdffCMj67AFBVF+NA1ixmMJ9jwUitf/s4LfOw9\nF1JVXhx2NBHJc6c9MzCzlcBCd18B3AjcNWyTu4BrgSuBq8xsyShtbgP+3d3fBGwH/jxY3uHuq1L+\nUyEYo2hREX/5riVcuayRnfs7+eK3NnKksyfsWCKS5zLpJloN/BDA3bcCdWZWA2Bm84Ej7r7H3ePA\n2mD7dG1WAQ8F+30YWDN+hyJDokVFfPCdi7nq0mb2Hz7BF7+1kX1tx8KOJSJ5LJNi0Ai0pTxuC5aN\ntK4VmD1Km0p37x22LUCZmd1nZk+Z2cezOwQZSVEkwh+/9VyuXTmfw529fOHejWzafijsWCKSpzIa\nMxgmMoZ1Iy1PXfYJ4FskxxeeMLMn3H1Duiepq6sgFoueNmg6DQ3VY26ba+Od7YZ3nc+C5un883d/\nzV0PvMgHr1nK769cQCQy2v/G3OcaL8qVnXzNBfmbrVByZVIMWnjtTACgieTg70jr5gTL+tK0OWZm\n5e7enbIt7v7VoQ3N7DHgfCBtMWhvP5FB7JE1NFTT1tY15va5lKts551Vw6euX85dD7zI3Q9vZvOO\nQ7z/aqO8NLO/BfL1NVOu7ORrLsjfbFMt12gFJJNuokeB6wDMbDnQ4u5dAO6+C6gxs3lmFgOuCbZP\n1+ZnJAebCX4+Ykn3mVkk2MeVwOasj1JGdc7sGj7zgUtZMKeGZ7cc5Lb/3sDeVo0jiEjSaYuBu68D\nNprZOpJXBd1sZjeY2R8Em9wEfAd4Erjf3beN1CbY9rPAB8zsSWA68N/u7sAe4DmSl5eudffnxu8Q\nZUhddSmf+rPlXP07zRw8coLbv7mBxzbuJZ5IhB1NREIWSUzCD4K2tq4xh87X0z6Y2GzPv9zG3T/a\nyvGeAZbMq+ODb19M/bSy0HNlQ7myk6+5IH+zTbVcDQ3VaQcLdQdygXrDwgZu/9BlXLCgni272vnM\n3c/yixf26SxBpECpGBSw2qpSbrnuAj749vNIJOCbjzhfvHcjuw/m319CIpJbKgYFLhKJ8KYLm/jC\nX1zOpefNZEdLJ7d9YwPfetTpPN4XdjwRmSAqBgIkB5dv+v1lfPw9FzKjtozHf72PT/3X0zz81E56\nNCW2yJSnYiCnWDa/ns9/6DKuf9siSmJFPPjkTv7ySz/jFy/sY2AwHnY8EcmRsdyBLFNcLFrE6ovP\n4opljfz42d38dMMevvmI8/BTu7j60mbefFETZSV664hMJTozkLTKS2P84Zvn81+3ruaqS5s50TPA\ndx/fzt/+xzoefOIVjSmITCH6805Oq35aOX+yeiHXXDGPx3+9l59t2MvD63bx42dfZfmiBlZdNAc7\nuzbr+Y5EJH+oGEjGqsqLedeV53D1pWfzq9/s5+fP7+O5ra08t7WVxukVrLyoicuXzGJaVWnYUUUk\nSyoGkrXSkiirLz6Lty6fw8t7O/jFC/vY8FIb9z++ne/9fDvnnV3HZUtmcbE1UFmmb1kTmQxUDGTM\nIpEIi5prWdRcy5+u7uOZLQd5bstBtr7aztZX27n3J87Sc6Zz0bkzuPDcGdRV64xBJF+pGMi4qK4o\n4W2XNPO2S5ppO9rNc1sP8uyWVl7ccZgXdxyGnzhnz6rionNnsPSc6Zwzu4ZYVNcviOQLFQMZdw21\n5bxzxTzeuWIerUe7eXH7ITZtP8RLu4+y++AxHnpqF6XFURaeNY3Fc+uws+s4e1aVioNIiFQMJKdm\n1paz5pJm1lzSTHfvAFt2tfPSq+1s3d3Ob3ce4bc7jwDJexvmNlYxf/Y05jfVsKCphvppZbpCSWSC\nqBjIhCkvjXGxNXCxNQDQcayXl3YfZdueo7zS0snOli527Os8uX11RTHNM6s4q6GKOQ2VNM+soqm+\nkpLisX/lqYiMTMVAQjOtqpTLlszisiWzAOjtH+TVA1280tLJKy0d7DrQxZZd7WzZ1X6yTSQCM+sq\naKwrZ+6caVSXxphVV87MunLqp5URLVJXk8hYqBhI3igtjp68OmlId+8A+w4dZ2/rMfa2HWNv23H2\ntR1j05ETbNpx+JT20aII9dPKaKgtp666lOnVpUyvKaOuuvTk4/LSmLqeREagYiB5rbw0xrlzpnHu\nnGmnLD/W3U8/EXznIVrbuzl4pJvWoydobe9mczAOMZLSkijTKkqoriimOvhZU1lCdXkx1ZXB8vIS\nqsqLKS+NUVYapUjFQwqAioFMSlXlxTQ0VFNX/vq3cG/fIEe6emjv6qW9q5cjwc/2zh6OdPXSeaKP\nwwd6GIyf/lvdIiQLUnlpjIqy4Oewx2UlUUpiRZQWRyktidLQdpye472UlESTy4qjlBRHKS0uoqRY\nxUXyk4qBTDmlJVFm11cyu74y7TaJRIITvQN0Hu+j60R/8F8fXSf66DzRz4mefk70DNDdO8CJ4L9D\nHd109w6ecb5YNEIsWhT8l/y9OFZ0cllxNEL0lGURiqNFyWXRIiJFyS6xoqJI8mckkvK4iKIIr61L\n2a52WgfHj/eeuj4SIRKJEIkkC9/J34f9HCpgRanLYcRtI5FIcgbMCKfuP2hD8FwEzwdQdqKP4z39\nBKuH1gbbnPr6DT2OEEndLGWf6dufvm3hFmoVAylIkUiEyrJiKsuKmV2febt4PEFP3wAnepIFort3\ngN7+QXr74/T2DdI3MEhxSYwjR7uD5YP0Bev7gt/7B+L0D8YZGEwwEPze293PQMoyfRN1fohEgAQp\nRWP0ApVmL+n3nVULiEaLuPGdi7m6oXq0JxwTFQORLBQVRagoK6ZilDmXGhqqaWsb+/dIJxIJBuOJ\nk8WhfyAe/J58HI8niCeSPwfjwc9hj4e2Obk+nqCispSOju6Tj4eeJxE8ZyKR8pOhx68tiyeSJSqe\nui2p25zaJh7sh8Tr2yQPlJO/l5RE6U35Rr3EsGqYCBYML5JD253c66k/Tm6QIP0+X7+v1xYUl8To\n6xsYtnzk5xxJulXDs2TWCqJFRdRUlozWeMxUDETyTCQSOdl9NJ7OtEjlUr5my9dcuaCLskVEJLMz\nAzO7E7ic5PnLLe6+PmXdGuAOYBBY6+63p2tjZs3AvUAU2A+8z917zex64KNAHPiau399vA5QRERO\n77RnBma2Eljo7iuAG4G7hm1yF3AtcCVwlZktGaXNbcC/u/ubgO3An5tZJfAZYA2wCviYmU0/4yMT\nEZGMZdJNtBr4IYC7bwXqzKyJR/V7AAAF50lEQVQGwMzmA0fcfY+7x4G1wfbp2qwCHgr2+zDJAnAZ\nsN7dO9y9G3iKZGEREZEJkkkxaATaUh63BctGWtcKzB6lTaW7955m26HlIiIyQcZyNdFoV9SmWzfS\n8my2PUVdXQWx2NhnrmzIwTW64yVfsylXdpQre/marVByZVIMWnjtTACgieTg70jr5gTL+tK0OWZm\n5UF30NC2I+3jmdECtbefyCD2yPL5UrF8zaZc2VGu7OVrtqmWa7QCkkk30aPAdQBmthxocfcuAHff\nBdSY2TwziwHXBNuna/MzkoPNBD8fAZ4FLjWzWjOrIjle8GSWxygiImcgMvwuvJGY2ZeAN5O89PNm\n4A1Ah7s/aGZvBv4+2PQBd//Hkdq4+yYzmw18EygDXgU+6O79ZnYd8LckL0P9V3f/9ngepIiIjC6j\nYiAiIlOb7kAWEREVAxERUTEQERFUDEREBBUDERFBxUBERCiwL7cZbSruELL8A/Amkv8Pvgi8C7gY\nOBxs8mV3/1EIuVYB3wc2B4t+A/wDI0w9PsG5bgTel7LoEmADUAkcD5b9L3ffOIGZlgH/A9zp7v+W\nL1O0p8l1D1AM9APvdfcDZtZPcmLIIavd/cy/5DnzXN9ghPd8Hrxe3wcagtXTSc6IcAfJfwtD7682\nd/+jHOca/hmxnhy+vwqmGKROq21mi4G7gRUhZXkLsCzIUg88DzwO/G93/39hZBrml+5+3dADM7uH\n5NTj3zezO4A/B/5zIgMFb/CvB3lWAu8BlpK8cfG3E5klyFAJ/CvwWMrioSnaT75OZvZNklO0/w7J\naVrWm9mD7n5kAnN9nuSHxPfM7Gbg48AnSd44uioXOTLMBcPe8ylT2of2eqV+yJvZ3cD/fW3VhL1e\nI31GPEYO31+F1E2UdiruEDwBDL3hjpL863bsM+/l3ipeP/V4mD4D3B5yhl7gHSTn1hqyivCnaB8p\n10eAB4Lf24D6HD5/OiPlGkk+vF4AmJkBte7+XA6fP52RPiNWkcP3V8GcGZCcDC+1C2FoWu3OiQ4S\nnIoPdW3cSPJ7IAaBvzKzj5Ocxvuv3P3QRGcLLDGzh0ieIn+OkaceD4WZXQrsCbo5AG4zsxnAVuCj\nwT+InHP3AWAgyDAk9CnaR8rl7scBzCxKcjqZ24JVZWZ2HzCX5FQyX5nIXIFT3vPkweuV4haSZw1D\nGs3sByQn3vz3XE6bk+Yz4upcvr8K6cxguNNOlZ1rZvZukv+j/4pkX+Ct7v5W4AXg70KK9TLJAvBu\n4AMku2ZS/2gI+3X7EPCN4Pd/Af7W3VPnzcoXY56iPReCQnAv8Li7D3WJfAL4MHAVcL2ZXTLBsTJ5\nz4f1epUAb3T3nweLDgP/B/hTkuN7twdzreU6R+pnRKpxf38V0pnBaFNxTzgzuxr4NPC77t7BqX2p\nDzHBffJD3H0fcH/wcIeZHSA5q+zwqcfDsgr4awB3fzBl+cPAH4cRKMW4TNGeI/cAL7v754YWuPtX\nh343s8eA80kOyk+IlKIEr73nf0B+vF4rgZPdQ8Gsy/cEDw+Z2QbgPHL4GTL8M8LMcvr+KqQzg7RT\ncU80M5sGfBm4Zmigx8weCL5GFJIfeBM+KBrkuN7MPhH83gjMIvmPYPjU42FkawKOuXufmUXM7Gdm\nVhusXkVIr1mKvJyiPbjapM/dP5uyzMzsvuB1jAW5NqfdSW5yjfSeD/31ClwKbBp6YGZvMbOvBL9X\nAhcB23L15CN9RpDj91dBzVo60rTaIeX4MMlT4tQ30z0kTwVPAMdIXiXTGkK2auA+oBYoIdll9Dwj\nTD0eQraLgc+7+9uDx+8BPkWyb3UfcKO7j/2bj7LP8k/APJKXa+4DrifZhRXaFO1pcs0EenhtfGyL\nu3/EzP4eeCvJfw8PufsXJjjXvwK3Muw9nwev1x+SfN//yt3vD7aLkbyqyEhe7PGf7n7PSPscp1wj\nfUZ8IMiQk/dXQRUDEREZWSF1E4mISBoqBiIiomIgIiIqBiIigoqBiIigYiAiIqgYiIgI8P8B0PnL\n/mqLXbgAAAAASUVORK5CYII=\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "metadata": {
        "id": "X9uiFJ-ogBRi",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "### Loading a Model"
      ]
    },
    {
      "metadata": {
        "id": "Doot7_0fY5tF",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "unlabeled_images_test = pd.read_csv('gdrive/My Drive/dataML/test.csv')\n",
        "#unlabeled_images_test = pd.read_csv('test.csv')\n",
        "\n",
        "X_unlabeled = unlabeled_images_test.values.reshape(unlabeled_images_test.shape[0],28,28,1)/255"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "dbtjpi8JgBRj",
        "colab_type": "code",
        "outputId": "ee65fc89-ef7e-4b99-b2bb-4d24d4e518c3",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "cell_type": "code",
      "source": [
        "with tf.Session() as sess:\n",
        "    \n",
        "    # Restore the model\n",
        "    saver.restore(sess, 'models_saving/my_model.ckpt')\n",
        "    \n",
        "\n",
        "    # Fetch Back Results\n",
        "    label = sess.run(Y, feed_dict={X:X_unlabeled,drop_rate:0})"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Restoring parameters from models_saving/my_model.ckpt\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "MZ1xS7oJZYkV",
        "colab_type": "code",
        "outputId": "6d483b80-a575-49e8-ce54-db9c91392100",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "cell_type": "code",
      "source": [
        "label"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([2, 0, 9, ..., 3, 9, 2])"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 74
        }
      ]
    },
    {
      "metadata": {
        "id": "ojRKNc76gBRx",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Predict the unlabeled test sets using the model"
      ]
    },
    {
      "metadata": {
        "id": "yQrGiou8gBRy",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "imageId = np.arange(1,label.shape[0]+1).tolist()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "9Oj02pY3gBR1",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "prediction_pd = pd.DataFrame({'ImageId':imageId, 'Label':label})"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "VLjMgeXEgBR4",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "prediction_pd.to_csv('gdrive/My Drive/dataML/out_cnn4.csv',sep = ',', index = False)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "ivz_HMLnZ684",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        ""
      ],
      "execution_count": 0,
      "outputs": []
    }
  ]
}
